metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <>
Subject Heterogeneous indexing batch size for different Metron feeds
Date Sun, 03 Dec 2017 12:27:42 GMT

We have noticed recently that no matter what batch size we use for Metron
indexing feeds, as long as we start using different batch size for
different Metron feeds, indexing topology throughput will start dropping
due to the high error rate! So I was wondering whether based on the current
indexing topology design, we have to choose the same batch size for all the
feeds or not. Otherwise, throughout will be dropped. I assume since it is
acceptable to use different batch sizes for different feeds, it is not
expected by design.

Moreover, I have noticed in practice that even if we change the batch size,
it will not affect the messages that are already in enrichments or indexing
topics, and it will only affect the new messages that are coming to the
parser. Therefore, we need to let all the messages pass the indexing
topology so that we can change the batch size!

It would be great if we can have more details regarding the design of this
section so we can understand our observations are based on the design or
some kind of bug.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message