nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nikinch <>
Subject Creating new linked entries in crawlDB
Date Thu, 11 Mar 2010 14:46:37 GMT

Hi everyone

I've been using nutch for a while now and i've come up on a snag.

I'm trying to find where new  linked pages are added to the segment as a
specific entry.
To make myself clear i've been through the fetch class and the crawlDBFilter
and reducer.
But i'm looking for the initial entry where, for a given page, the links are
transformed into segment entries, my objective here is to pass down te
initial inject url  to all it's liked pages. So when i create an entry for
the linked urls of a wegbpage i'll add metadata to their definition giving
them this originating url.
By the time i get to CrawlDBFilter i already have entries for linked pages
and lost the notion of which seed url brought us here.
I thought the job would be done in the Fetcher maybe in the output function
but i'm not finding where it happens. So if anyone knows and could point me
in the right direction i'd appreciate it.

View this message in context:
Sent from the Nutch - Dev mailing list archive at

View raw message