nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2287) Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
Date Tue, 05 Jul 2016 22:39:11 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363402#comment-15363402
] 

ASF GitHub Bot commented on NUTCH-2287:
---------------------------------------

GitHub user naegelejd opened a pull request:

    https://github.com/apache/nutch/pull/131

    NUTCH-2287 Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/naegelejd/nutch NUTCH-2287

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #131
    
----
commit 0fff24acff99df1ce9d3f9c9b1a2c674f9c8e325
Author: Joseph Naegele <jnaegele@grierforensics.com>
Date:   2016-06-30T17:42:34Z

    NUTCH-2287 Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy

----


> Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
> -------------------------------------------------------------------------------
>
>                 Key: NUTCH-2287
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2287
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer, plugin
>    Affects Versions: 1.12
>            Reporter: Joseph Naegele
>
> Elasticsearch's API (since at least v2.0) includes the {{BulkProcessor}}, which automatically
handles flushing bulk requests given a max doc count and/or max bulk size. It also now (I
believe since 2.2.0) offers a {{BackoffPolicy}} option, allowing the BulkProcessor/Client
to retry bulk requests when the Elasticsearch cluster is saturated. Using the {{BulkProcessor}}
was originally suggested [here|https://issues.apache.org/jira/browse/NUTCH-1527?focusedCommentId=13666616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13666616].
> Refactoring the {{indexer-elastic}} plugin to use the {{BulkProcessor}} will greatly
simplify the existing plugin at the cost of slightly less debug logging. Additionally, it
will allow the plugin to handle cluster saturation gracefully (rather than raising a RuntimeException
and killing the reduce task), by using a configurable "exponential back-off policy".
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.3/java-docs-bulk-processor.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message