nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santiago PĂ©rez <elara...@gmail.com>
Subject Increasing the score of especific pages
Date Thu, 11 Mar 2010 10:24:10 GMT

I want to modify Nutch for increasing the score of some pages in their
CrawlDatum. The objective of this is recognizing which pages include a
certain token. Increasing the score to a high value will be useful for being
chosen again in the next Segment generation. 

I modified like this:

Fetcher.java:
(...)
case ProtocolStatus.SUCCESS:        // got a page
            	  content.setContent((Integer.toString(0)).getBytes());
            	  if (tokenIncluded) {
                      float score = Float.valueOf( "500.0" );
                      fit.datum.setScore(score);
                  }
(...)

After fetching, when the crawldb has to be updated, the entries of the MR
update proccess are the fetched urls. Each fetched one appeared twice like
this:

http://bardeportes.blogspot.com/
Version: 7
Status: 2 (db_fetched)
Fetch time: Thu Mar 11 11:06:09 CET 2010
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 1.0
Signature: null
Metadata: _pst_: success(1), lastModified=0

...

http://bardeportes.blogspot.com/
Version: 7
Status: 33 (fetch_success)
Fetch time: Thu Mar 11 11:06:17 CET 2010
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 500.0
Signature: null
Metadata: _ngt_: 1268301973666_pst_: success(1), lastModified=0


And the final score introduce to crawldb is 1.0 (never 500.0)
Any idea?

(I hope someone understands the issue)
-- 
View this message in context: http://old.nabble.com/Increasing-the-score-of-especific-pages-tp27861656p27861656.html
Sent from the Nutch - Dev mailing list archive at Nabble.com.


Mime
View raw message