metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <>
Subject Re: Heterogeneous indexing batch size for different Metron feeds
Date Tue, 05 Dec 2017 00:34:18 GMT
Any thoughts?

On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian <> wrote:

> Hi,
> We have noticed recently that no matter what batch size we use for Metron
> indexing feeds, as long as we start using different batch size for
> different Metron feeds, indexing topology throughput will start dropping
> due to the high error rate! So I was wondering whether based on the current
> indexing topology design, we have to choose the same batch size for all the
> feeds or not. Otherwise, throughout will be dropped. I assume since it is
> acceptable to use different batch sizes for different feeds, it is not
> expected by design.
> Moreover, I have noticed in practice that even if we change the batch
> size, it will not affect the messages that are already in enrichments or
> indexing topics, and it will only affect the new messages that are coming
> to the parser. Therefore, we need to let all the messages pass the indexing
> topology so that we can change the batch size!
> It would be great if we can have more details regarding the design of this
> section so we can understand our observations are based on the design or
> some kind of bug.
> Regards,
> Ali


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message