lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <>
Subject Re: Retrieving Documents
Date Mon, 19 Dec 2011 16:37:00 GMT
"A programmer has a problem. She tried to solve it with regular expressions.
Now she has two problems"....

You could *try* PatternReplaceCharFilterFactory. Note that this is
applied to the entire input string *before* tokenization. I'm thinking
you could write a clever regex that transformed everything except
your page number into "".

But I think a better thing to explore is for you to tell us how you're
getting the epub format in the first place. If you're programmatically
splitting it up (say in SolrJ), just find the interesting bit of data
in the SolrJ program and add it as a field in the document.

Or, if you're using DIH... Or....

The interesting bit will be several fold:
1> how grouping the documents such that they are a single
    entity, i.e. book. Will you have one large document or
    one document for each page?
2> How will you handle phrase searches when the phrase
    begins on one page and ends on the next?

This is one of those things that looks easy at the start but
quickly gets kinda ugly. Good luck!


On Mon, Dec 19, 2011 at 10:29 AM, Dan McGinn-Combs <> wrote:
> I can see why you are confused. Re-reading it, I'm confused.
> Here's my dilemna.
> I am trying index some one hundred or so books all in EPUB format. The
> goal is to provide research functions, i.e. people who need to
> reference specific quotes, pages and books for their writing.
> I don't know if EPUB is designed to do this by default, but each book
> is created/converted to EPUB using Calibre. Each page is packed into
> the EPUB file as a separate HTML file with the format
> <title>_split_<page number>.html. So the upshot of my question is
> whether there is a way to extract the page number from the title of
> the embedded HTML page and expose that in a Solr field that I can
> subsequently display to the user?
> I hope that makes a bit more sense.
> Still looking through the Wiki because it seems to be stuffed with goodies.
> Dan
> On Sat, Dec 17, 2011 at 2:59 PM, Otis Gospodnetic
> <> wrote:
>> Hi Dan,
>> I don't follow the second paragraph.  Not sure what you are trying to do, what you've
tried, what didn't work and how...
>> Otis
>> ----
>> Performance Monitoring SaaS for Solr -
>>> From: Dan McGinn-Combs <>
>>>To: "" <>
>>>Sent: Saturday, December 17, 2011 9:30 AM
>>>Subject: Re: Retrieving Documents
>>>Good pointer. Thank you, that is exactly what I had in mind. To the
>>>second point, yes, sort of.
>>>I've managed to take apart a sample of the ePub documents (there are a
>>>finite number). Inside the ePub are single HTML documents that are a
>>>single page of the overall book. It would be super to be able to parse
>>>the title (originally formed from the page number) to set up a
>>>dynamically generated documented and include that as part of the
>>>results. Combing the wiki now since that's where every answers seems
>>>to be! Pointers welcome though.
>>>Dan McGinn-Combs
>>>On Dec 16, 2011, at 11:52 PM, Otis Gospodnetic
>>><> wrote:
>>>> Hi Dan,
>>>> 1) Are you looking for
>>>> 2) Hundreds of words in a field should not be a problem for highlighting. 
But it sounds like this long field may contain content that corresponds to N different pages
in a publication and you would like to inform the searcher which page the match was on, and
not just that a match was somewhere in that big piece of text.  One way to deal with that
is to break your document into N smaller documents - one document for each page.
>>>> Otis
>>>> ----
>>>> Performance Monitoring SaaS for Solr -
>>>>> ________________________________
>>>>> From: Dan McGinn-Combs <>
>>>>> To:
>>>>> Sent: Friday, December 16, 2011 4:33 PM
>>>>> Subject: Retrieving Documents
>>>>> I've been doing a fair amount of reading and experimenting with Solr
>>>>> lately. I find that it does a good job of indexing very structured
>>>>> documents. However, the application I have in mind is build around
>>>>> long EPUB documents.
>>>>> Of course, I found the Extract components useful for indexing the
>>>>> EPUBs. However, I would like to be able to
>>>>> * Size the "highlight" portion of text around the query parameters
>>>>> (i.e. show 20 or 30 words) and
>>>>> * Retrieve a location within the document so I can display that "page"
>>>>> from the EPUB.
>>>>> What is common practice for these? I notice that if I have a list of
>>>>> (short) text segments in fields, they are stored without too much fuss
>>>>> and are retrievable. However, I'm talking about a field of potentially
>>>>> hundreds of words.
>>>>> Thanks for any pointers,
>>>>> Dan
>>>>> --
>>>>> Dan McGinn-Combs
>>>>> Peachtree City, Georgia USA
> --
> Dan McGinn-Combs
> Google Voice: +1 404 492 7532
> Peachtree City, Georgia USA

View raw message