Skip to content

Commit 1fc9638

Browse files
yoffyoff
committed
Python: port redos .qhelp from js
1 parent 0cf9c95 commit 1fc9638

File tree

4 files changed

+212
-0
lines changed

4 files changed

+212
-0
lines changed

‎python/ql/src/Security/CWE-730/PolynomialReDoS.qhelp‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
<!DOCTYPE qhelp PUBLIC
2+
"-//Semmle//qhelp//EN"
3+
"qhelp.dtd">
4+
5+
<qhelp>
6+
7+
<include src="ReDoSIntroduction.inc.qhelp" />
8+
9+
<example>
10+
<p>
11+
12+
Consider this use of a regular expression, which removes
13+
all leading and trailing whitespace in a string:
14+
15+
</p>
16+
17+
<sample language="python">
18+
re.sub(r"^\s+|\s+$", "", text) # BAD
19+
</sample>
20+
21+
<p>
22+
23+
The sub-expression <code>"\s+$"</code> will match the
24+
whitespace characters in <code>text</code> from left to right, but it
25+
can start matching anywhere within a whitespace sequence. This is
26+
problematic for strings that do <strong>not</strong> end with a whitespace
27+
character. Such a string will force the regular expression engine to
28+
process each whitespace sequence once per whitespace character in the
29+
sequence.
30+
31+
</p>
32+
33+
<p>
34+
35+
This ultimately means that the time cost of trimming a
36+
string is quadratic in the length of the string. So a string like
37+
<code>"a b"</code> will take milliseconds to process, but a similar
38+
string with a million spaces instead of just one will take several
39+
minutes.
40+
41+
</p>
42+
43+
<p>
44+
45+
Avoid this problem by rewriting the regular expression to
46+
not contain the ambiguity about when to start matching whitespace
47+
sequences. For instance, by using a negative look-behind
48+
(<code>^\s+|(?&lt;!\s)\s+$</code>), or just by using the built-in strip
49+
method (<code>text.strip()</code>).
50+
51+
</p>
52+
53+
<p>
54+
55+
Note that the sub-expression <code>"^\s+"</code> is
56+
<strong>not</strong> problematic as the <code>^</code> anchor restricts
57+
when that sub-expression can start matching, and as the regular
58+
expression engine matches from left to right.
59+
60+
</p>
61+
62+
</example>
63+
64+
<example>
65+
66+
<p>
67+
68+
As a similar, but slightly subtler problem, consider the
69+
regular expression that matches lines with numbers, possibly written
70+
using scientific notation:
71+
</p>
72+
73+
<sample language="python">
74+
^0\.\d+E?\d+$ # BAD
75+
</sample>
76+
77+
<p>
78+
79+
The problem with this regular expression is in the
80+
sub-expression <code>\d+E?\d+</code> because the second
81+
<code>\d+</code> can start matching digits anywhere after the first
82+
match of the first <code>\d+</code> if there is no <code>E</code> in
83+
the input string.
84+
85+
</p>
86+
87+
<p>
88+
89+
This is problematic for strings that do <strong>not</strong>
90+
end with a digit. Such a string will force the regular expression
91+
engine to process each digit sequence once per digit in the sequence,
92+
again leading to a quadratic time complexity.
93+
94+
</p>
95+
96+
<p>
97+
98+
To make the processing faster, the regular expression
99+
should be rewritten such that the two <code>\d+</code> sub-expressions
100+
do not have overlapping matches: <code>^0\.\d+(E\d+)?$</code>.
101+
102+
</p>
103+
104+
</example>
105+
106+
<include src="ReDoSReferences.inc.qhelp"/>
107+
108+
</qhelp>

‎python/ql/src/Security/CWE-730/ReDoS.qhelp‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
<!DOCTYPE qhelp PUBLIC
2+
"-//Semmle//qhelp//EN"
3+
"qhelp.dtd">
4+
5+
<qhelp>
6+
7+
<include src="ReDoSIntroduction.inc.qhelp" />
8+
9+
<example>
10+
<p>
11+
Consider this regular expression:
12+
</p>
13+
<sample language="python">
14+
^_(__|.)+_$
15+
</sample>
16+
<p>
17+
Its sub-expression <code>"(__|.)+?"</code> can match the string <code>"__"</code> either by the
18+
first alternative <code>"__"</code> to the left of the <code>"|"</code> operator, or by two
19+
repetitions of the second alternative <code>"."</code> to the right. Thus, a string consisting
20+
of an odd number of underscores followed by some other character will cause the regular
21+
expression engine to run for an exponential amount of time before rejecting the input.
22+
</p>
23+
<p>
24+
This problem can be avoided by rewriting the regular expression to remove the ambiguity between
25+
the two branches of the alternative inside the repetition:
26+
</p>
27+
<sample language="python">
28+
^_(__|[^_])+_$
29+
</sample>
30+
</example>
31+
32+
<include src="ReDoSReferences.inc.qhelp"/>
33+
34+
</qhelp>

‎python/ql/src/Security/CWE-730/ReDoSIntroduction.inc.qhelp‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
<!DOCTYPE qhelp PUBLIC
2+
"-//Semmle//qhelp//EN"
3+
"qhelp.dtd">
4+
<qhelp>
5+
<overview>
6+
<p>
7+
8+
Some regular expressions take a long time to match certain
9+
input strings to the point where the time it takes to match a string
10+
of length <i>n</i> is proportional to <i>n<sup>k</sup></i> or even
11+
<i>2<sup>n</sup></i>. Such regular expressions can negatively affect
12+
performance, or even allow a malicious user to perform a Denial of
13+
Service ("DoS") attack by crafting an expensive input string for the
14+
regular expression to match.
15+
16+
</p>
17+
18+
<p>
19+
20+
The regular expression engine provided by Python uses a backtracking non-deterministic finite
21+
automata to implement regular expression matching. While this approach
22+
is space-efficient and allows supporting advanced features like
23+
capture groups, it is not time-efficient in general. The worst-case
24+
time complexity of such an automaton can be polynomial or even
25+
exponential, meaning that for strings of a certain shape, increasing
26+
the input length by ten characters may make the automaton about 1000
27+
times slower.
28+
29+
</p>
30+
31+
<p>
32+
33+
Typically, a regular expression is affected by this
34+
problem if it contains a repetition of the form <code>r*</code> or
35+
<code>r+</code> where the sub-expression <code>r</code> is ambiguous
36+
in the sense that it can match some string in multiple ways. More
37+
information about the precise circumstances can be found in the
38+
references.
39+
40+
</p>
41+
</overview>
42+
43+
<recommendation>
44+
45+
<p>
46+
47+
Modify the regular expression to remove the ambiguity, or
48+
ensure that the strings matched with the regular expression are short
49+
enough that the time-complexity does not matter.
50+
51+
</p>
52+
53+
</recommendation>
54+
</qhelp>

‎python/ql/src/Security/CWE-730/ReDoSReferences.inc.qhelp‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
<!DOCTYPE qhelp PUBLIC
2+
"-//Semmle//qhelp//EN"
3+
"qhelp.dtd">
4+
<qhelp>
5+
<references>
6+
<li>
7+
OWASP:
8+
<a href="https://www.owasp.org/index.php/Regular_expression_Denial_of_Service_-_ReDoS">Regular expression Denial of Service - ReDoS</a>.
9+
</li>
10+
<li>Wikipedia: <a href="https://en.wikipedia.org/wiki/ReDoS">ReDoS</a>.</li>
11+
<li>Wikipedia: <a href="https://en.wikipedia.org/wiki/Time_complexity">Time complexity</a>.</li>
12+
<li>James Kirrage, Asiri Rathnayake, Hayo Thielecke:
13+
<a href="http://www.cs.bham.ac.uk/~hxt/research/reg-exp-sec.pdf">Static Analysis for Regular Expression Denial-of-Service Attack</a>.
14+
</li>
15+
</references>
16+
</qhelp>

0 commit comments

Comments
 (0)