lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-11916) new SortableTextField using docValues built from the original string input
Date Fri, 26 Jan 2018 23:06:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-11916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341765#comment-16341765
] 

Hoss Man commented on SOLR-11916:
---------------------------------

Hmmm... good point.

I thought that since this new type didn't go out of it's way to enable "useDocValuesAsStored"
that it was a non-issue and the docValues would _never_ be used in place of a stored value
(but even if that were true, we should definitely have a test proving it)

Skimming the relevant code now I realize that as a FieldProperty, the schemaVersion is the
only thing that drives the (default) value of USE_DOCVALUES_AS_STORED regardless of the FieldType
impl – so you are absolutely correct, we need to do "something" in {{SortableTextField}}
to account for this propery.
{quote}Perhaps this fieldType overrides useDocValuesAsStored() to see if maxChars=-1 (no limit)
so it can vary it's output based on that?
{quote}
 
 Hmm ... My concern with that approach is that it might be confusing the users how explicitly
set {{useDocValuesAsStored="true" stored="false"}} on a fieldType (or field) – perhaps w/o
even being aware of the default maxChars safety valve – and then don't understand why they
aren't getting any values back?

One possibility would be be for {{SortableTextField.init}} to override the (implicit) default
{{useDocValuesAsStored=(schemaVersion>1.6)}} with it's own default based on {{useDocValuesAsStored=(maxChars==-1)}}
_and_ fail with a server error (on init) if a configuration includes an explicit {{useDocValuesAsStored=true
maxChars="anything other then -1"}} ?
----
Personally, my vote would be – at least initially – to just say "useDocValuesAsStored
is not supported for SortableTextField", set the default appropriately & fail on init
if anyone tries to explicitly set it to "true" .... but since FieldProperites can be set on
both the {{fieldType}} and the {{field}} I don't think it would even be possible for a fieldType
to *stop* someone from creating a {{<field type="some_sortable_text" useDocValuesAsStored="true"}}
... is it?

> new SortableTextField using docValues built from the original string input
> --------------------------------------------------------------------------
>
>                 Key: SOLR-11916
>                 URL: https://issues.apache.org/jira/browse/SOLR-11916
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Hoss Man
>            Assignee: Hoss Man
>            Priority: Major
>         Attachments: SOLR-11916.patch
>
>
> I propose adding a new SortableTextField subclass that would functionally work the same
as TextField except:
> * {{docValues="true|false"}} could be configured, with the default being "true"
> * The docValues would contain the original input values (just like StrField) for sorting
(or faceting)
> ** By default, to protect users from excessively large docValues, only the first 1024
of each field value would be used -- but this could be overridden with configuration.
> ----
> Consider the following sample configuration:
> {code}
> <field name="title" type="text_sortable" docValues="true"
>        indexed="true" docValues="true" stored="true" multiValued="false"/>
> <fieldType name="text_sortable" class="solr.SortableTextField">
>   <analyzer type="index">
>    ...
>   </analyzer>
>   <analyzer type="query">
>    ...
>   </analyzer>
> </fieldType>
> {code}
> Given a document with a title of "Solr In Action"
> Users could:
> * Search for individual (indexed) terms in the "title" field: {{q=title:solr}}
> * Sort documents by title ( {{sort=title asc}} ) such that this document's sort value
would be "Solr In Action"
> If another document had a "title" value that was longer then 1024 chars, then the docValues
would be built using only the first 1024 characters of the value (unless the user modified
the configuration) 
> This would be functionally equivalent to the following existing configuration - including
the on disk index segments - except that the on disk DocValues would refer directly to the
"title" field, reducing the total number of "field infos" in the index (which has a small
impact on segment housekeeping and merge times) and end users would not need to sort on an
alternate "title_string" field name - the original "title" field name would always be used
directly.
> {code}
> <field name="title" type="text"
>        indexed="true" docValues="true" stored="true" multiValued="false"/>
> <field name="title_string" type="string"
>        indexed="false" docValues="true" stored="false" multiValued="false"/>
> <copyField source="title" dest="title_string" maxChars="1024" />
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message