metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mattf-horton <...@git.apache.org>
Subject [GitHub] metron issue #481: METRON-322 Global Batching and Flushing
Date Mon, 10 Jul 2017 05:36:37 GMT
Github user mattf-horton commented on the issue:

    https://github.com/apache/metron/pull/481
  
    @cestella said:
    ```
    Since this is a core change, it bears probably developing an acceptance test plan. 
    Something like creating a dummy sensor, creating a batch size of 10 and sending 
    in 5 messages and ensuring things get flushed.
    
    Also, it might be worth adjusting the IndexingIntegrationTest to set the hdfs writer 
    config in metron-integration-test/src/main/config/indexing/test.json to a large batch

    size (say 50) and a timeout of, say, 5 seconds to ensure that batches get flushed as 
    part of the integration tests.
    ```
    Unfortunately a lot of existing tests have fairly sensitive dependence on hdfs parameters
in indexing/test.json, so tweaking it arbitrarily is not feasible.  However, I was able to
enlarge the elasticsearch values, and I have mocked procedures in the unit test that test
the following cases:
    
    1. Test that flush on batchSize continues to work correctly, without time-based triggers
going off.
    1. Test that flush occurs on less-than-full queue, when a message comes in and the queue
is older than batchTimeout.  (Age of the queue is determined as currentTime() - timeOfEnqueingFirstMsg().)
    1. Test that flush occurs on less-than-full queue, when a TickTuple comes in and the queue
is older than batchTimeout.
    1. Test that default batchTimeout is correctly calculated as (TOPOLOGY_MESSAGE_TIMEOUT_SECS
/ 2 / batchTimeoutDivisor - 1).
    1. Test that when default batchTimeout is smaller than configured batchTimeout, that the
default is used.
    1. Test that when default batchTimeout is greater than configured batchTimeout, that the
configured batchTimeout is used.
    
    All of the above test cases are tested, with mocked writers and a FakeClock, in the BulkMessageWriterBoltTest
unit test file, with the latest commit above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message