nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney (JIRA) <>
Subject [jira] Commented: (NUTCH-547) Redirection handling: YahooSlurp's algorithm
Date Wed, 07 Nov 2007 20:05:51 GMT


Doğacan Güney commented on NUTCH-547:

Dennis, thanks for the update and testing.  I have been a bit unresponsive lately, I hope
that this will change (very) soon.

I actually was going to commit it a while back, but Andrzej said that he is working on adding
alias support/etc. (as outlined in his earlier email to nutch-dev) which included this patch
as part of his effort but it probably fell off his radar during his move and everything.

Anyway, I feel that lately we have been too reluctant to commit new stuff. So I am also +1
for committing it. If we break something or miss something, we can always fix it later :)

So I am going to give this one a day or two (say, until sunday) and commit it if there are
no objections. Is this OK with you, Dennis?

> Redirection handling: YahooSlurp's algorithm
> --------------------------------------------
>                 Key: NUTCH-547
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Doğacan Güney
>             Fix For: 1.0.0
>         Attachments: NUTCH-547-3.patch, redirect_draft.patch, redirect_draft_v2.patch
> After reading Yahoo's algorithm (then one Andrzej linked to:
> )
> in the redirect/alias handling discussion, I had a bit of a spare
> time, so I implemented it.
> Note that the patch I am attaching is for the 'choosing' algorithm described in
> Yahoo's help page. It makes no attempt to handle aliases in any way. (See
for the discussion about alias handling).
> E.g,
> generate ""
> fetch "http:/" which redirects to
> "".
> Update second page's datum's metadata to indicate that
> "" is the representative form.
> Updatedb, invertlinks, etc...
> While indexing second page, change its "url" field to
> "".

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message