metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Sumbul <>
Subject Architectural reason to split in 4 topologies / impact on the kafka ressources
Date Fri, 22 Jun 2018 21:25:50 GMT
Hi Everyone,

I was asking myself what was the architectural reason to split the
ingestion in metron in 4 differents toppologies that all read/write to

For example, why the parsing and enrichment topologies have not been
merged? Would it not be possible when you parse the message to directly
enricht it?

Im asking that because splitting in several topologies means that all of
the topologies read/write to Kafka, which produce a bigger load on the
kafka cluster and then a need for way more infrastructure/servers. The cost
is especially true when we speak about TBs of data ingested every day.

Im sure there were a very good reason, I was just curious.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message