gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexis (JIRA)" <j...@apache.org>
Subject [jira] Created: (GORA-20) Flush datastore regularly
Date Sat, 18 Dec 2010 18:53:01 GMT
Flush datastore regularly

                 Key: GORA-20
                 URL: https://issues.apache.org/jira/browse/GORA-20
             Project: Gora
          Issue Type: New Feature
          Components: storage
            Reporter: Alexis

Right now you need to explicitly call the flush method to make the IO operation happen, or
close the datastore.

The issue is described here: http://techvineyard.blogspot.com/2010/12/build-nutch-20.html#Free_up_the_memory.
Click on the image to see it in real size and look at the Heap utilization on the top right

Not everybody has infinite memory. In a Nutch fetch process, I usually run into trouble after
around 20k urls downloaded because it takes up all the memory, the Java Heap space being set
to 1G with a system that "only" has 1G RAM as well.

The feature consists of allowing the datastore to be flushed regularly during the Hadoop job's
reducer, org.apache.gora.mapreduce.GoraReducer. We would just add a maxBuffer parameter, which
default value is 10000 for example and that you can override in org.apache.gora.mapreduce.GoraOutputFormat.
It indicates the maximum number of records buffered in memory before the next flush operation
occurs to actually write them in the datastore. This would actually be a member of the org.apache.hadoop.mapreduce.RecordWriter
extension returned by getRecordWriter method.

An idea of the fix is suggested in the above link. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message