metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (METRON-329) Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing
Date Fri, 03 Mar 2017 00:51:45 GMT

     [ https://issues.apache.org/jira/browse/METRON-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Foley updated METRON-329:
------------------------------
    Description: 
Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing.
This primarily affects:
* BulkMessageWriterBolt
* BulkWriterComponent
* IndexingConfigurations
* and newly added utilities in BatchTimeoutHelper

All of the other changes (from METRON-516) are basically boilerplate, just laying "getBatchTimeout()"
facade methods alongside existing "getBatchSize()" facade methods. (There's a lot of layers!)

The flush-on-timeout logic is fairly straightforward. It was implemented by a refactoring
of BulkWriterComponent and basic Tick Tuple reception in BulkMessageWriterBolt.

The tricky part was figuring out the appropriate setting for 'topology.tick.tuple.freq.secs'
if the administrator configures non-default batchTimeouts. It is necessary to enumerate the
batchTimeout settings for all configured sensorNames, which is implemented in IndexingConfigurations$getAllConfiguredTimeouts().
Then multiple other factors must be taken into account to determine the allowed and recommended
settings, which is implemented in BatchTimeoutHelper. If there are better ways to accomplish
these things, please share your ideas.

  was:
Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing.
This primarily affects:
* BulkMessageWriterBolt
* BulkWriterComponent
* IndexingConfigurations
* and newly added utilities in BatchTimeoutHelper


> Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing
> --------------------------------------------------------------------------------------------
>
>                 Key: METRON-329
>                 URL: https://issues.apache.org/jira/browse/METRON-329
>             Project: Metron
>          Issue Type: Sub-task
>            Reporter: Ajay Yadav
>            Assignee: Matt Foley
>
> Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing.
> This primarily affects:
> * BulkMessageWriterBolt
> * BulkWriterComponent
> * IndexingConfigurations
> * and newly added utilities in BatchTimeoutHelper
> All of the other changes (from METRON-516) are basically boilerplate, just laying "getBatchTimeout()"
facade methods alongside existing "getBatchSize()" facade methods. (There's a lot of layers!)
> The flush-on-timeout logic is fairly straightforward. It was implemented by a refactoring
of BulkWriterComponent and basic Tick Tuple reception in BulkMessageWriterBolt.
> The tricky part was figuring out the appropriate setting for 'topology.tick.tuple.freq.secs'
if the administrator configures non-default batchTimeouts. It is necessary to enumerate the
batchTimeout settings for all configured sensorNames, which is implemented in IndexingConfigurations$getAllConfiguredTimeouts().
Then multiple other factors must be taken into account to determine the allowed and recommended
settings, which is implemented in BatchTimeoutHelper. If there are better ways to accomplish
these things, please share your ideas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message