lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Sokolov <>
Subject Re: Best document format / markup for text indexing?
Date Wed, 23 Nov 2011 15:01:43 GMT
In my experience, books and other semi-structured text documents are 
best handled as XML.  There are many many different XML "vocabularies" 
for doing this, each of which has benefits for different kinds of 
documents.  You probably should look at TEI, NLM Book, and DocBook 
though - these are some widely-used standard formats for capturing 
structured book-type texts.  There are other standards for journal 
articles and other kinds of documents.

The question of how to store, index and retrieve the kind of information 
and structure captured by XML documents has gotten a lot of attention, 
too.  There are XML-specific data stores such as MarkLogic and eXist 
(which uses Lucene for full text search).  Or you could consider 
"rolling your own" with something like Solr/Lucene as a search index.  
Because you're posting on this list, I assume you're considering the 
last option, which is a good one, but will require some development 
effort as you consider how to map document structures into indexes, how 
to preserve document structure when you highlight query terms, etc.

-Mike Sokolov

PS - if you are interested in professional help, please consider our 
platform ( and drop me an e-mail.

On 11/17/2011 3:46 PM, logic.cpp wrote:
> tl;dr version:
> We're converting tons (hundreds of thousands?) of books into digital text.
> What is the best format/markup/ebook standard/document standard/other to use for easiest
and best text search support?
> ***
> Longer version;
> The following are some desired user experience features of the project, these probably
influence the way in which the content should preferably be stored;
> - Granular access to the text content.
> Users would be able to fetch a specific phrase in a specific paragraph in a specific
page in a specific chapter in a specific book. (A 'document' may consist of a single chapter
of a book).
> - Cross referencing.
> Most likely achieved through a RDBMS, users should have references to/from content that
refers or mentions a topic or quotes related content in other books.
> (Similar to Wikipedia articles linking to one-another.)
> - Full text search
> This is probably where Lucene comes in.
> So which format/markup/standard would allow for software to easily fetch and cross-reference
granular bits of data, as well as be easily indexable by Lucene?
> Would it maybe be better to store all the books' digital text straight into the RDBMS?
In which case, can Lucene index such data?
> Thanks
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message