lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shay Hummel <shay.hum...@gmail.com>
Subject Re: Text dependent analyzer
Date Mon, 20 Apr 2015 14:29:48 GMT
Hi Rich

Thank you very much,
I understand your solution and will try to do something in that spirit.

Shay

On Fri, Apr 17, 2015 at 8:35 PM Rich Cariens <richcariens@gmail.com> wrote:

> Ahoy, ahoy!
>
> I was playing around with something similar for indexing multi-lingual
> documents, Shay. The code is up on github
> <https://github.com/whateverdood/cross-lingual-search> and needs
> attention, but you're welcome to see if anything in there helps. The basic
> idea is this:
>
>    1. A custom CharFilter uses the ICU4J sentence BreakIterator to get
>    sentences out of the char stream.
>       1. Each sentence is lang-id'd using the cybozu Detector, and a
>       thread-local (ugh)
>       2. A ThreadLocal (ugh) is updated to with languages and their
>       offsets (where a run of a particular language ends)
>    2. A custom Filter then marks each token with it's language (relying
>    on that ThreadLocal) if possible so the next custom Filter
>    3. ...checks the tokens language and recruits the appropriate stemmer.
>    4. Other Filters like ICUFoldingFilter kick in to do their thing,
>
> Does this help at all?
>
> On Fri, Apr 17, 2015 at 1:06 PM, Benson Margulies <benson@basistech.com>
> wrote:
>
>> If you wait tokenization to depend on sentences, and you insist on
>> being inside Lucene, you have to be a Tokenizer. Your tokenizer can
>> set an attribute on the token that ends a sentence. Then, downstream,
>> filters can  read-ahead tokens to get the full sentence and buffer
>> tokens as needed.
>>
>>
>>
>> On Fri, Apr 17, 2015 at 1:00 PM, Ahmet Arslan <iorixxx@yahoo.com.invalid>
>> wrote:
>> > Hi Hummel,
>> >
>> > There was an effort to bring open-nlp capabilities to Lucene:
>> > https://issues.apache.org/jira/browse/LUCENE-2899
>> >
>> > Lance was working on it to keep it up-to-date. But, it looks like it is
>> not always best to accomplish all things inside Lucene.
>> > I personally would do the sentence detection outside of the Lucene.
>> >
>> > By the way, I remember there was a way to consume all upstream token
>> stream.
>> >
>> > I think it was consuming all input and injecting one concatenated huge
>> term/token.
>> >
>> > KeywordTokenizer has similar behaviour. It injects a single token.
>> >
>> http://lucene.apache.org/core/3_0_3/api/all/org/apache/lucene/analysis/KeywordAnalyzer.html
>> >
>> > Ahmet
>> >
>> >
>> > On Wednesday, April 15, 2015 3:12 PM, Shay Hummel <
>> shay.hummel@gmail.com> wrote:
>> > Hi Ahment,
>> > Thank you for the reply,
>> > That's exactly what I am doing. At the moment, to index a document, I
>> break
>> > it to sentences, and each sentence is analyzed (lemmatizing, stopword
>> > removal etc.)
>> > Now, what I am looking for is a way to create an analyzer (a class which
>> > extends lucene's analyzer). This analyzer will be used for index and
>> query
>> > processing. It (a like the english analyzer) will receive the text and
>> > produce tokens.
>> > The Api of Analyzer requires implementing the createComponents which
>> > is not dependent
>> > on the text being analyzed. This fact is problematic since as you know
>> the
>> > OpenNlp sentence breaking depends on the text it gets (OpenNlp uses the
>> > model files to provide spans of each sentence and then break them).
>> > Is there a way around it?
>> >
>> > Shay
>> >
>> >
>> > On Wed, Apr 15, 2015 at 3:50 AM Ahmet Arslan <iorixxx@yahoo.com.invalid
>> >
>> > wrote:
>> >
>> >> Hi Hummel,
>> >>
>> >> You can perform sentence detection outside of the solr, using opennlp
>> for
>> >> instance, and then feed them to solr.
>> >>
>> >>
>> https://opennlp.apache.org/documentation/1.5.2-incubating/manual/opennlp.html#tools.sentdetect
>> >>
>> >> Ahmet
>> >>
>> >>
>> >>
>> >>
>> >> On Tuesday, April 14, 2015 8:12 PM, Shay Hummel <shay.hummel@gmail.com
>> >
>> >> wrote:
>> >> Hi
>> >> I would like to create a text dependent analyzer.
>> >> That is, *given a string*, the analyzer will:
>> >> 1. Read the entire text and break it into sentences.
>> >> 2. Each sentence will then be tokenized, possesive removal, lowercased,
>> >> mark terms and stemmed.
>> >>
>> >> The second part is essentially what happens in english analyzer
>> >> (createComponent). However, this is not dependent of the text it
>> receives -
>> >> which is the first part of what I am trying to do.
>> >>
>> >> So ... How can it be achieved?
>> >>
>> >> Thank you,
>> >>
>> >> Shay Hummel
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message