nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bruce" <bedoug...@earthlink.net>
Subject nutch/lucene question..
Date Wed, 02 Aug 2006 13:41:46 GMT
hi.

i have a possible project where i'm looking at extracting information from
various public/college websites. i don't need to index the text/content of
the sites. i do need to extract specific information.

as an example, a site might have a course schedule page, which in turn has
links to the departments page, which in turn has links to the class
information page. from a tree structure this would be:
  course listings by semester
     departments for the semester
       classes of each department
          class information

obviously nutch/lucene has the ability to crawl a given site, does it have
the ability to somehow 'link'/maintain a given relationship to the upstream
page for a given piece of information.

for my needs, i need to maintain the semester i get, as i follow the "link
to the department, etc... this approach allows me to then store the complete
course information in a db, so i can then iterate through the course
information.

i can accomplish this now, by creating a unique crawling app for each
school. my curiousity is whether nutch/lucene can provide a basic crawling
engine that i could then plug into, for my specific needs. i'm also curious
as to the amount of additional development that would have to be created for
my needs...

thanks

-bruce


Mime
View raw message