lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark harwood <>
Subject Re: Soliciting Design Thoughts on Date Searching
Date Thu, 01 Mar 2007 12:13:27 GMT
GATE is the other entity extraction framework ( and comes out of the box
with a lot of this stuff.

Even once you've parsed the dates your next problem is representing and querying time - you
referred to the fact that documents could represent single dates, multiple dates or time lines.
I've found that a very flexible form of representing and querying time is to use the spatial
technology normally used in GIS (geographic informations systems). They've got the smarts
for representing shapes such as points, lines, multi-lines, multi-points  etc and the retrieval
tools that compare query shapes with data shapes using operators like "contains", "intersects",
"within" etc. This spatial technology can be applied to handling time as well - you just make
use of one dimension instead of the usual two.


----- Original Message ----
From: Otis Gospodnetic <>
Sent: Thursday, 1 March, 2007 10:25:24 AM
Subject: Re: Soliciting Design Thoughts on Date Searching

Ah, I once worked in a place where we did exactly that - recognition and extraction of useful
nuggets from emails - dates, emails, URLs, attachments, people, places...see for
the next generation of that.  I believe Zoe subsequently did something very similar.  I think
Zoe is still free, so you might be able to get ideas from its source code.

There may also be some UIMA components for date extraction.  UIMA is now in ASF's Incubator.

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy --  -  Tag  -  Search  -  Share

----- Original Message ----
From: Steven Parkes <>
Sent: Wednesday, February 28, 2007 6:56:02 PM
Subject: RE: Soliciting Design Thoughts on Date Searching

Yeah, date finding is a little like entity extraction, since dates can
have many formats, depending on how crazy you want to get ("a week from
tomorrow" is 3/8/2007 if you know that this e-mail was written today).
So much so that I went and looked up lingpipe, but they seem to not be
concerned with dates. 

Even if you don't get crazy, it's not straightforward: is 3/8/2007 March
8th or August 3rd? Dates can be written many ways. The real challenge is
recognizing dates.

As Chris said, once you have them, you just stick them in the token
stream. In fact, you can emit the date token (as Chris suggested, with
some delimiter that helps you know it's a date) with a position
increment of zero and then emit the regular tokens so that the token
stream will have both and aligned.

-----Original Message-----
From: Chris Hostetter [] 
Sent: Wednesday, February 28, 2007 3:26 PM
To: Lucene Users
Subject: Re: Soliciting Design Thoughts on Date Searching

: I have generic material that _contain_ dates: historic time lines,
: certificates, news articles, forms, deeds, testimonies, and wildly
: free form genealogical information.  The dates have no specific
: structure, obvious context, nor consistency.

identifying an extracting dates from bulk text sounds like a pretyt
interesting analysys problem ... if you wrote a Tokenizer that could
recognize dates, you could then format them using something like
to ensure it would be easy to find them ... but Lucene Analyzers can not
currently create terns in multiple fields - so if you wanted a special
"date" field for each doc, you would have to extract those dates in a
preprocessing step.

if you aren't picky how your index is stored however, there is no reason
why you can't have a single field with your "text" terms and your "date"
terms ... you would just have to be careful to know the differnece in
searching ... make your analyzer prefix all of your date terms with
soemthing it would never let your regular terms start with (ie "__")
make sure you bear that structure in mind when creating your RangeFilter
on dates.

: Now this is where my personal knowledge of Lucene breaks down.
: Assuming I can extract each date from a source's body and convert it
: to a usable format, can a Lucene Date Field hold more than one date?

fields can contain as many values as you want -- or none at all.

: If the answer is yes, problem solved!  I'll just pile on a ton of

definitely yes.


To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

New Yahoo! Mail is the ultimate force in competitive emailing. Find out more at the Yahoo!
Mail Championships. Plus: play games and win prizes.* 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message