metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cestella <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #424: METRON-672: SolrIndexingIntegrationTest ...
Date Thu, 26 Jan 2017 00:12:24 GMT
GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/424

    METRON-672: SolrIndexingIntegrationTest fails intermittently

    This failure is due to a change in default behavior when indexing was split off into a
separate configuration file.  The default batch size was changed from `5` to `1` in particular.
 This, by itself, is not a problem, but the `IndexingIntegrationTest` (base class for Solr
and Elastic search integration tests):
    * submits the configs
    * starts the indexing topology
    * writes the input data
    
    The writing of the input data may happen before the topology fully loads or the configuration
fully loads, especially if the machine running the unit tests is under load (like with travis).
 As a result, the first record may end up with the default batch size (of 1) and write out
immediately because the indexing configs haven't loaded into zookeeper just yet.  In that
circumstance, eventually the configs load and the batch size is set to `5`.  Meanwhile we've
written 10 records and are expecting 10 in return, but because you wrote the first out already
and then the next 5, we have another 4 pending to be written by the `BulkMessageWriterBolt`.
    
    So, the failure scenario is as follows:
    * Message 1 is received and the indexing config hasn't loaded yet, so the batch size is
1 and it immediately gets written out
    * Message 2 - 5 are each received and the indexing config has loaded, so the batch size
is 5 and it queues up
    * Message 6 is received and the batch writes out
    * Messages 7 - 10 are received, but never make a full batch, so we time out waiting for
them to write out
    
    The fix is to ensure that we don't write out messages to kafka until the configs are loaded,
which is what this PR does.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron METRON-672

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #424
    
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message