nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <>
Subject [jira] Commented: (NUTCH-395) Increase fetching speed
Date Mon, 13 Nov 2006 09:59:38 GMT
    [ ] 
Andrzej Bialecki  commented on NUTCH-395:

+1 - this patch looks good to me - if you could just fix the whitespace issues prior to committing,
so that it conforms to the coding style ...

> Increase fetching speed
> -----------------------
>                 Key: NUTCH-395
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.9.0, 0.8.1
>            Reporter: Sami Siren
>         Assigned To: Sami Siren
>         Attachments: nutch-0.8-performance.txt, NUTCH-395-trunk-metadata-only-2.patch,
> There have been some discussion on nutch mailing lists about fetcher being slow, this
patch tried to address that. the patch is just a quich hack and needs some cleaning up, it
also currently applies to 0.8 branch and not trunk and it has also not been tested in large.
What it changes?
> Metadata - the original metadata uses spellchecking, new version does not (a decorator
is provided that can do it and it should perhaps be used where http headers are handled but
in most of the cases the functionality is not required)
> Reading/writing various data structures - patch tries to do io more efficiently see the
patch for details.
> Initial benchmark:
> A small benchmark was done to measure the performance of changes with a script that basically
does the following:
> -inject a list of urls into a fresh crawldb
> -create fetchlist (10k urls pointing to local filesystem)
> -fetch
> -updatedb
> original code from 0.8-branch:
> real    10m51.907s
> user    10m9.914s
> sys     0m21.285s
> after applying the patch
> real    4m15.313s
> user    3m42.598s
> sys     0m18.485s

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:


View raw message