lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7256) PatternReplaceCharFilter can make Lucene hang
Date Thu, 09 Jun 2016 13:28:21 GMT


Robert Muir commented on LUCENE-7256:

I don't think we should do that, it won't help. Nor should we offer apis in lucene that pretend
to take timeouts (like ExitableDirectoryReader). Especially in this case where it will not
work. I hate that its tests are time-based and fail sporatically.

There is nothing we can do to fix things like this with java's regex stuff. We can't protect
against the user being stupid:

> PatternReplaceCharFilter can make Lucene hang
> ---------------------------------------------
>                 Key: LUCENE-7256
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: modules/analysis
>    Affects Versions: 5.4.1
>         Environment: alpine linux v3.3
>            Reporter: Tom Fotherby
>            Priority: Minor
> I'm using ElasticSearch (v2.2.0 , Lucene v5.4.1) and it's [Pattern Replace Char Filter|]
(Lucenes PatternReplaceCharFilter) . I need to filter out urls from my query text before it
is tokenised. But I found that some input strings cause ElasticSearch to "hang" (slowly eating
more CPU and memory) until the system crashes.
> ----
> *Example*
> {code}
> // Character filters are used to "tidy up" a string *before* it is tokenized.
> 'char_filter' => [
>     'url_removal_pattern' => [
>         'type'        => 'pattern_replace',
>         'pattern'     => '(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?«»""'']))',
>         'replacement' => '',
>     ],
> {code}
> This filter was working fine for some weeks until suddenly ElasticSearch started crashing.
We found someone was trying to do a javascript injection attack in our search box.
> I pasted the regex and the attack string into 
> * Regexp: 
>  * {code}(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s!()\[\]{};:\'".,<>?«»""''])){code}
> * Test string: 
>  * {code}\";fjs.parentNode.insertBefore(js,fjs);}}(document,\"script\",\"twitter-wjs\"{code}
> shows the problem to be "Catastrophic backtracking"
> bq. Catastrophic backtracking has been detected and the execution of your expression
has been halted. To find out more what this is, please read the following article: [Runaway
Regular Expressions|].
> It would be great if Lucene could detect "Catastrophic backtracking" and throw a error
or return null.
> ----
> As an aside, I created a unit test for our PHP application that uses the same regexp
and test string. (PHP can understand the same regexp, even though it's obviously for Java
in the ElasticSearch case) . Interestingly in php, the regex results in `null` which is the
documented response of [preg_replace|] when
a error occurs. If PHP can return a error rather than crashing - surely Lucene / Java can
too :trollface: ?
> {code}
> namespace app\tests\unit;
> use \yii\codeception\TestCase;
> class TagsControllerTest extends TestCase
> {
>     public function testRegexForURLDetection()
>     {
>         $regex = '(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?«»""'']))';
>         // Test the Catastrophic backtracking problem
>         $testString = "\";fjs.parentNode.insertBefore(js,fjs);}}(document,\"script\",\"twitter-wjs\"";
>         // This shows the regex is not working for our test string - it gives null but
should give 'hello '
>         $this->assertEquals(null, preg_replace("/$regex/", '', "hello $testString"));
>     }
> }
> {code}
> ----
> (I originally [opened a ticket|]
to the ElasticSearch project but got told opening it here would be more appropriate - sorry
if I'm wrong)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message