nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <>
Subject [jira] Commented: (NUTCH-799) SOLRIndexer to commit once all reducers have finished
Date Fri, 05 Mar 2010 09:56:27 GMT


Andrzej Bialecki  commented on NUTCH-799:

I think it's ok to do it this way - the commit per reducer may be actually harmful if commit
succeeds but the task is killed for any reason and re-ran.

Note: the patch has some formatting errors.

> SOLRIndexer to commit once all reducers have finished
> -----------------------------------------------------
>                 Key: NUTCH-799
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer
>            Reporter: Julien Nioche
>             Fix For: 1.1
>         Attachments: NUTCH-799.patch
> What about doing only one SOLR commit after the MR job has finished in SOLRIndexer instead
of doing that at the end of every Reducer? 
> I ran into timeout exceptions in some of my reducers and I suspect that this was due
to the fact that other reducers had already finished and called commit. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message