lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-6780) some param values are duplicated when they override defaults, or are combined with appends values, or are an invariant that overrides a request param
Date Wed, 04 Feb 2015 16:59:37 GMT

     [ https://issues.apache.org/jira/browse/SOLR-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hoss Man updated SOLR-6780:
---------------------------
      Description: 
The {{DefaultSolrParams}} class, which is used as the basis for the implementation of {{defaults}},
{{appends}} and {{invariants}} params had a bug in it's implementation of {{getParameterNamesIterator()}}
that could result in the same param key being returned multiple times.

In many code paths of Solr, this bug had no effects -- but in other cases, it resulted in
code which iterated over the list of all parameters to take action multiple times for the
(valid) key=value pairs.

There were 4 main areas where this bug had unexpected & problematic behavior for end users:

{panel:title=main problem areas & impacts}
* ExtractingRequestHandler
** "literal.\*" params will be duplicated if overridden by defaults/invariants/appends - this
will result in redundent literal field=value params being added to the document.
** impact: multiple values in literal fields when not expected/desired
* FacetComponent
** "facet.\*" params will be duplicated if overridden by defaults/invariants/appends - this
can result in redundent computation and identical facet.field, facet.query, or facet.range
blocks in the response
** impact: wasted computation & increased response size
* SpellCheckComponent
** when "custom params" (ie: "spellcheck.\[dictionary name\].XXXX=YYYY" are used in used in
defaults, appends, or invariants, it can cause redudent XXXXX=YYYY params to be used.
** when "spellcheck.collateParam.XXXX=YYYY" type params are used defaults, appends, or invariants,
it can cause redundent XXXX=YYYY params to exist in the collation verification queries.
** impact: unclear to me at first glance, probably just wasted computation & increased
response size
* AnalyticsComponent
** "olap.\*" params will be duplicated if overridden by defaults/invariants/appends - this
can result in redundent computation
** impact: unclear to me at first glance, probably just wasted computation & increased
response size
{panel}

Other less serious impacts were redundent values in "echoParams" as well as some small amounts
of wasted computation in other code paths that iterated over the set of params (due to a slightly
larger set of param values)



{panel:title=Original bug report: "Merging request parameters with defaults produce duplicate
entries"}


When a parameter (e.g. echoParams) is specified and overrides the default on the handler,
it actually generates two entries for that key with the same value. 

Most of the time it is just a confusion and not an issue, however, some components will do
the work twice. For example faceting component as described in http://search-lucene.com/m/QTPaSlFUQ1/duplicate

It may also be connected to SOLR-6369

The cause seems to be the interplay between *DefaultSolrParams#getParameterNamesIterator()*
which just returns param names in sequence and *SolrParams#toNamedList()* which uses the first
(override then default) value for each key, without deduplication.

It's easily reproducible in trunk against schemaless example with 
bq. curl "http://localhost:8983/solr/schemaless/select?indent=true&echoParams=all"

I've also spot checked it and it seems to be reproducible back to Solr 4.1.

{panel}

  was:

The {{DefaultSolrParams}} class, which is used as the basis for the implementation of {{defaults}},
{{appends}} and {{invariants}} params had a bug in it's implementation of {{getParameterNamesIterator()}}
that could result in the same param key being returned multiple times.

In many code paths of Solr, this bug had no effects -- but in other cases, it resulted in
code which iterated over the list of all parameters to take action multiple times for the
(valid) key=value pairs.

There were 4 main areas where this bug had unexpected & problematic behavior for end users:

{panel:title=main problem areas & impacts}
* ExtractingRequestHandler
** "literal.\*" params will be duplicated if overridden by defaults/invariants/appends - this
will result in redundent literal field=value params being added to the document.
** impact: multiple values in literal fields when not expected/desired
* FacetComponent
** "facet.\*" params will be duplicated if overridden by defaults/invariants/appends - this
can result in redundent computation and identical facet.field, facet.query, or facet.range
blocks in the response
** impact: wasted computation & increased response size
* SpellCheckComponent
** when "custom params" (ie: "spellcheck.\[dictionary name\].XXXX=YYYY" are used in used in
defaults, appends, or invariants, it can cause redudent XXXXX=YYYY params to be used.
** when "spellcheck.collateParam.XXXX=YYYY" type params are used defaults, appends, or invariants,
it can cause redundent XXXX=YYYY params to exist in the collation verification queries.
** impact: unclear to me at first glance, probably just wasted computation & increased
response size
* AnalyticsComponent
** "olap.\*" params will be duplicated if overridden by defaults/invariants/appends - this
can result in redundent computation
** impact: unclear to me at first glance, probably just wasted computation & increased
response size
{panel}

Other less serious impacts were redundent values in "echoParams" as well as some small amounts
of wasted computation in other code paths that iterated over the set of params (due to a slightly
larger set of param values)



{panel:title=Original bug report: "Merging request parameters with defaults produce duplicate
entries"}


When a parameter (e.g. echoParams) is specified and overrides the default on the handler,
it actually generates two entries for that key with the same value. 

Most of the time it is just a confusion and not an issue, however, some components will do
the work twice. For example faceting component as described in http://search-lucene.com/m/QTPaSlFUQ1/duplicate

It may also be connected to SOLR-6369

The cause seems to be the interplay between *DefaultSolrParams#getParameterNamesIterator()*
which just returns param names in sequence and *SolrParams#toNamedList()* which uses the first
(override then default) value for each key, without deduplication.

It's easily reproducible in trunk against schemaless example with 
bq. curl "http://localhost:8983/solr/schemaless/select?indent=true&echoParams=all"

I've also spot checked it and it seems to be reproducible back to Solr 4.1.

{panel}

    Fix Version/s:     (was: 4.10.3)
                   4.10.4

> some param values are duplicated when they override defaults, or are combined with appends
values, or are an invariant that overrides a request param
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-6780
>                 URL: https://issues.apache.org/jira/browse/SOLR-6780
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 4.1, 5.0, Trunk
>            Reporter: Alexandre Rafalovitch
>            Assignee: Hoss Man
>              Labels: parameters
>             Fix For: 4.10.4, 5.0, Trunk
>
>         Attachments: SOLR-6780.patch
>
>
> The {{DefaultSolrParams}} class, which is used as the basis for the implementation of
{{defaults}}, {{appends}} and {{invariants}} params had a bug in it's implementation of {{getParameterNamesIterator()}}
that could result in the same param key being returned multiple times.
> In many code paths of Solr, this bug had no effects -- but in other cases, it resulted
in code which iterated over the list of all parameters to take action multiple times for the
(valid) key=value pairs.
> There were 4 main areas where this bug had unexpected & problematic behavior for
end users:
> {panel:title=main problem areas & impacts}
> * ExtractingRequestHandler
> ** "literal.\*" params will be duplicated if overridden by defaults/invariants/appends
- this will result in redundent literal field=value params being added to the document.
> ** impact: multiple values in literal fields when not expected/desired
> * FacetComponent
> ** "facet.\*" params will be duplicated if overridden by defaults/invariants/appends
- this can result in redundent computation and identical facet.field, facet.query, or facet.range
blocks in the response
> ** impact: wasted computation & increased response size
> * SpellCheckComponent
> ** when "custom params" (ie: "spellcheck.\[dictionary name\].XXXX=YYYY" are used in used
in defaults, appends, or invariants, it can cause redudent XXXXX=YYYY params to be used.
> ** when "spellcheck.collateParam.XXXX=YYYY" type params are used defaults, appends, or
invariants, it can cause redundent XXXX=YYYY params to exist in the collation verification
queries.
> ** impact: unclear to me at first glance, probably just wasted computation & increased
response size
> * AnalyticsComponent
> ** "olap.\*" params will be duplicated if overridden by defaults/invariants/appends -
this can result in redundent computation
> ** impact: unclear to me at first glance, probably just wasted computation & increased
response size
> {panel}
> Other less serious impacts were redundent values in "echoParams" as well as some small
amounts of wasted computation in other code paths that iterated over the set of params (due
to a slightly larger set of param values)
> {panel:title=Original bug report: "Merging request parameters with defaults produce duplicate
entries"}
> When a parameter (e.g. echoParams) is specified and overrides the default on the handler,
it actually generates two entries for that key with the same value. 
> Most of the time it is just a confusion and not an issue, however, some components will
do the work twice. For example faceting component as described in http://search-lucene.com/m/QTPaSlFUQ1/duplicate
> It may also be connected to SOLR-6369
> The cause seems to be the interplay between *DefaultSolrParams#getParameterNamesIterator()*
which just returns param names in sequence and *SolrParams#toNamedList()* which uses the first
(override then default) value for each key, without deduplication.
> It's easily reproducible in trunk against schemaless example with 
> bq. curl "http://localhost:8983/solr/schemaless/select?indent=true&echoParams=all"
> I've also spot checked it and it seems to be reproducible back to Solr 4.1.
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message