lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julie Tibshirani <juliet...@gmail.com>
Subject Re: ANN search current state
Date Wed, 22 Jul 2020 17:54:13 GMT
Tomoko -- this fits with my experience as well. I really like the idea of treating LUCENE-9322
as setting up a framework for experimentation + benchmarking (but not requiring us to commit
a particular ANN implementation quite yet).

Julie

On 2020/07/17 12:16:18, Tomoko Uchida <t...@gmail.com> wrote: 
> > would it make sense to create a separate Lucene module for ANN search ?> 
> 
> From a bit of my experience with LUCENE-9004, it is currently impossible to> 
> plug in or opt in custom codecs and indexing chain for aknn search without> 
> touching lucene-core module (plz correct that if it's wrong).> 
> I think LUCENE-9322 (unified low-level Codec/Format for dense vectors)> 
> would open the possibility for us to experiment different aknn algorithms> 
> on some sandbox modules or even separated jars from Lucene itself.> 
> 
> Tomoko> 
> 
> 
> 2020年7月17日(金) 17:00 Tommaso Teofili <to...@gmail.com>:> 
> 
> > would it make sense to create a separate Lucene module for ANN search ?> 
> > we could then experiment with the different approaches and compare them> 
> > across the same benchmarks.> 
> >> 
> > On Thu, 16 Jul 2020 at 23:14, Ali Akhtar <al...@ali.actor> wrote:> 
> >> 
> > > I’m a bit of a layman in this area, but if we are talking about formats>

> > for> 
> > > vectors, I vote for the one used by FastAI word vectors. It’s pretty easy>

> > > to work with.> 
> > >> 
> > > If we are talking about the same / similiar things, if not just ignore me>

> > > 😀> 
> > >> 
> > > On Thu, 16 Jul 2020 at 7:06 PM, Michael Sokolov <ms...@gmail.com>>

> > > wrote:> 
> > >> 
> > > > We have some prototype implementations in the issues you found.  If>

> > > > you want to try out the approaches in those issues, you could build>

> > > > Lucene from source and patch it, but there is no release containing>

> > > > KNN/vector support. We're still working to establish consensus on what>

> > > > the best way forward is. I think the most fruitful thing we can do at>

> > > > the moment is establish a format for storing and accessing vectors>

> > > > that will support different approaches since there is such a rich>

> > > > variety of algorithms and approaches in this area. The last issue you>

> > > > pointed to is focused on the format.> 
> > > >> 
> > > > On Wed, Jul 15, 2020 at 11:20 AM Alex K <ak...@gmail.com> wrote:>

> > > > >> 
> > > > > Hi Mikhail,> 
> > > > >> 
> > > > > I'm not sure about the state of ANN in lucene proper. Very interested>

> > > to> 
> > > > > see the response from others.> 
> > > > > I've been doing some work on ANN for an Elasticsearch plugin:>

> > > > > http://elastiknn.klibisz.com/> 
> > > > > I think it's possible to extract my custom queries and modeling code>

> > so> 
> > > > > that it's elasticsearch-agnostic and can be used directly in Lucene>

> > > apps.> 
> > > > > However I'm much more familiar with Elasticsearch's APIs and>

> > > > usage/testing> 
> > > > > patterns than I am with raw Lucene, so I'd likely need to get some>

> > help> 
> > > > > from the Lucene community.> 
> > > > > Please LMK if that sounds interesting to anyone.> 
> > > > >> 
> > > > > - Alex> 
> > > > >> 
> > > > >> 
> > > > >> 
> > > > > On Wed, Jul 15, 2020 at 11:11 AM Mikhail <wm...@mail.ru.invalid>>

> > > > wrote:> 
> > > > >> 
> > > > > >> 
> > > > > > Hi,> 
> > > > > >> 
> > > > > >            I want to incorporate semantic search in my project,>

> > which> 
> > > > uses> 
> > > > > > Lucene. I want to use sentence embeddings and ANN (approximate>

> > > nearest> 
> > > > > > neighbor) search. I found the related Lucene issues:> 
> > > > > > https://issues.apache.org/jira/browse/LUCENE-9004 ,> 
> > > > > > https://issues.apache.org/jira/browse/LUCENE-9136 ,> 
> > > > > > https://issues.apache.org/jira/browse/LUCENE-9322 . I see that>

> > there> 
> > > > > > are some related work and related PRs. What is the current state
of> 
> > > > this> 
> > > > > > functionality?> 
> > > > > >> 
> > > > > > --> 
> > > > > > Thanks,> 
> > > > > > Mikhail> 
> > > > > >> 
> > > > > >> 
> > > >> 
> > > > --------------------------------------------------------------------->

> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org> 
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org>

> > > >> 
> > > >> 
> > >> 
> >> 
> 
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message