nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Nagel (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2011) Endpoint to support realtime JSON output from the fetcher
Date Mon, 18 May 2015 08:00:00 GMT


Sebastian Nagel commented on NUTCH-2011:

Yes, that's because of the nodeDB feature (the crawl has run before without any problems).
Of course, a parsing fetcher needs more heap memory, esp., if there are huge files to parse.
I traced the memory usage of the running fetch process: it started with 350 MB and reached
1 MB after 35.000 documents fetched. After that the process slowed down, I had to kill it
after 39.000 fetched docs when the fetch speed went down to 0.1 docs / sec. while the process
used more and more CPU to run the garbage collector.
Roughly, 16kB are required for one FetchNode entry. Seems to be a lot at a first glance, but
in fact: URL, title, outlinks + anchors take some space (including the overhead of Java objects).
Also, a hash map <Integer, FetchNode> is not the ideal structure to hold a list of consecutively
enumerated elements.

> Endpoint to support realtime JSON output from the fetcher
> ---------------------------------------------------------
>                 Key: NUTCH-2011
>                 URL:
>             Project: Nutch
>          Issue Type: Sub-task
>          Components: fetcher, REST_api
>            Reporter: Sujen Shah
>            Assignee: Chris A. Mattmann
>              Labels: memex
>             Fix For: 1.11
> This fix will create an endpoint to query the Nutch REST service and get a real-time
JSON response of the current/past Fetched URLs. 
> This endpoint also includes pagination of the output to reduce data transfer bw in large

This message was sent by Atlassian JIRA

View raw message