lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Federico Pici <akira...@libero.it.INVALID>
Subject Re: About custom score using Solr8/Lucene8
Date Thu, 09 May 2019 16:04:39 GMT
Yes i saw it. But it allows me to give a different ranking (and a different order) for the
top N documents that match a search.

We would like to give documents an order in absence of a query.

to be more precise if do something like /solr/collection_1/select/?q=*.*

i normally wouldn’t get a score.

Using my (hopefully) custom score i would be able to give documents a “value” or “score”
(based on factors that are outside search terms).

For example with a /solr/collection_1/select/?q=*.* 
first documents could be the ones a user viewed most and that contains a certain quantity
of information(so not strictly related to a search)

My strategy must be applied to all documents (and not the first one matched by a query and
re-ranked) 


> Il giorno 9 mag 2019, alle ore 17:40, Diego Ceccarelli (BLOOMBERG/ LONDON) <dceccarelli4@bloomberg.net>
ha scritto:
> 
> Have you seen the rerank plugin [1] ? maybe you could apply your custom scoring function
only to the top documents? 
> 
> [1]  https://lucene.apache.org/solr/6_6_0//solr-core/org/apache/solr/search/ReRankQParserPlugin.html
> 
> From: java-user@lucene.apache.org At: 05/09/19 16:24:07To:  java-user@lucene.apache.org
> Subject: Re: About custom score using Solr8/Lucene8
> 
> Diego it could be a nice workaround in case i was not able to perform custom 
> scoring (or in case custom scoring function would generate a slow query)
> Thanks
> 
> I still hope i will be able to implement a custom score function in some way 
> 
> I was almost able to achieve such result with ValueSourceParser but i’m not 
> sure is the best practice
> 
>> Il giorno 9 mag 2019, alle ore 16:02, Alessandro Benedetti 
> <benedetti.alex85@gmail.com> ha scritto:
>> 
>> What about external fields?
>> 
> https://lucene.apache.org/solr/guide/6_6/working-with-external-files-and-process
> es.html
>> I admit I have not read in details all the thread, but from a superficial
>> reading, it could help.
>> Apologies if it's not helpful enough!
>> 
>> Cheers
>> 
>> 
>> On Thu, May 9, 2019 at 12:57 PM Federico Pici <akirapix@libero.it.invalid>
>> wrote:
>> 
>>> Yes this was an option, and it solves a part of the problem.
>>> 
>>> But let’s consider this use case:
>>> 
>>> for each user index we need to calculate some important informations based
>>> on the most relevant terms, the “starred” documents, the number of times
>>> the were opened by user (and other key factors based on documents usage).
>>> So, maybe weekly, we should update this “global” factors that would affect
>>> each user’s document score. If this week i opened, for example, more
>>> documents of typeA and less of typeB, ALL documents of typeA would be more
>>> important
>>> of all documents of typeB (this is just a strategy). So each time i
>>> recalculate “global” importance key factors i would need to update every
>>> user’s document to change its score.
>>> 
>>> In this case i would fear scaling up, because as users grow up (and as
>>> each user’s document number grow up) i would been updating more and more
>>> document.
>>> 
>>> That’s why i would like to separate this moments. I would calculate
>>> “global” key factors in a separate process, but i would like to use them
at
>>> query time to generate different scoring strate
>>> 
>>> 
>>> 
>>> 
>>>> Il giorno 9 mag 2019, alle ore 13:29, Diego Ceccarelli (BLOOMBERG/
>>> LONDON) <dceccarelli4@bloomberg.net> ha scritto:
>>>> 
>>>> why would you need to reindex all the users documents everynight? is the
>>> score affected by the number of documents per user?
>>>> 
>>>> Also for scores, you might be able to use in-place updates:
>>>> 
>>>> 
>>> 
> https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#Updati
> ngPartsofDocuments-In-PlaceUpdates
>>>> 
>>>> so no need to reindex all the document.
>>>> 
>>>> 
>>>> From: java-user@lucene.apache.org At: 05/09/19 11:07:37To:  Diego
>>> Ceccarelli (BLOOMBERG/ LONDON ) ,  java-user@lucene.apache.org
>>>> Subject: Re: About custom score using Solr8/Lucene8
>>>> 
>>>> Well,
>>>> 
>>>> my index is basically a list of documents of users. so i could say i got
>>> a
>>>> couple of millions of documents per user.
>>>> 
>>>> I need to produce a score that is unrelated with search term (something
>>> like
>>>> “give me documents that are the most important to user”). I would
>>>> search for documents of a user without specifying a particular term.
>>> Documents
>>>> that are the most important for a user are calculated with strategy
>>> based
>>>> on particular text analysis (we still do not have a strategy).
>>>> 
>>>> I evaluated the idea of storing a score for each document based on an
>>> external
>>>> scoring strategy (a job that reindex each user’s document every night?),
>>> but
>>>> i do not know how could scale up (more user with many documents could
>>> make this
>>>> strategy problematic in the long time), so the idea was to be able
>>>> to calculate this scoring (or other scoring strategy) trying to follow
>>> some
>>>> best-practice (and as i can see using CustomScoreQuery is the most
>>> preferred
>>>> way)
>>>> 
>>>> I was not able to try a solution because of Lucene8 missing
>>> CustomScoreQuery.
>>>> I made something very basic using ValueSourceParser (but is this a good
>>>> practice?)
>>>> 
>>>> 
>>>>> Il giorno 9 mag 2019, alle ore 10:50, Diego Ceccarelli (BLOOMBERG/
>>> LONDON)
>>>> <dceccarelli4@bloomberg.net> ha scritto:
>>>>> 
>>>>> Hi Federico,
>>>>> 
>>>>> Could you give more details on what is the score that you want to
>>> produce?
>>>> how big is the collection?
>>>>> Is the score affected by the query? if not, would be possible to store
>>> the
>>>> score with the document?
>>>>> 
>>>>> Cheers,
>>>>> Diego
>>>>> 
>>>>> From: java-user@lucene.apache.org At: 05/09/19 08:13:35To:
>>>> java-user@lucene.apache.org
>>>>> Subject: Re: About custom score using Solr8/Lucene8
>>>>> 
>>>>> Hello Mikhail,
>>>>> yes i did. I read all the page at
>>>>> https://issues.apache.org/jira/browse/LUCENE-8099
>>>>> <https://issues.apache.org/jira/browse/LUCENE-8099>
>>>>> 
>>>>> I still do not understand where/how to start to write a class to be
>>> embedded
>>>> in
>>>>> solr8. All
>>>>> available example since 2012 where written around
>>>>> CustomScoreQuery/CustomScoreProvider.
>>>>> 
>>>>> I really don’t know how to setup a minimal project to implement custom
>>>> scoring
>>>>> in solr8/lucene8 using FunctionScoreQuery
>>>>> 
>>>>>> Il giorno 8 mag 2019, alle ore 22:15, Mikhail Khludnev <
>>> mkhl@apache.org> ha
>>>>> scritto:
>>>>>> 
>>>>>> Hello.
>>>>>> 
>>>>>> Have you checked the fourth item at
>>>>>> http://lucene.apache.org/core/8_0_0/MIGRATE.html ?
>>>>>> 
>>>>>> On Wed, May 8, 2019 at 5:46 PM Federico Pici
>>> <akirapix@libero.it.invalid>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> i was trying to implement custom score inside my solr8 instance.
>>>>>>> On google i found some examples, all focused on using
>>> CustomScoreProvider
>>>>>>> and CustomScoreQuery classes. This classes have been removed
in
>>>>>>> Solr8/Lucene8
>>>>>>> and, from Lucene 8 changeling, i saw a developer /contributor
saying
>>> to
>>>>>>> use FunctionScoreQuery.
>>>>>>> 
>>>>>>> Googling around was unsuccessful as i found no example, so i
have no
>>> clue
>>>>>>> from where to start from
>>>>>>> 
>>>>>>> I think i have a very basic need. Generate a score from field
in
>>>>>>> documents using logic completely unrelated with query itself.
>>>>>>> 
>>>>>>> The only way available (until any clue…) is using
>>>>>>> org.apache.solr.search.ValueSourceParser
>>>>>>> 
>>>>>>> 
>>>>>>> Is there any example, at least to understand how to start a minimal
>>> basic
>>>>>>> project?
>>>>>>> 
>>>>>>> Thanks
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Sincerely yours
>>>>>> Mikhail Khludnev
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>> 
>>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> 
>>> 
>> 
>> -- 
>> --------------------------
>> 
>> Benedetti Alessandro
>> Visiting card - http://about.me/alessandro_benedetti
>> Blog - http://alexbenedetti.blogspot.co.uk
>> 
>> "Tyger, tyger burning bright
>> In the forests of the night,
>> What immortal hand or eye
>> Could frame thy fearful symmetry?"
>> 
>> William Blake - Songs of Experience -1794 England
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message