metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <>
Subject [DISCUSS] Opinionated Data Flows
Date Thu, 06 Oct 2016 16:52:03 GMT
I totally "bird dogged the previous thread" as Casey likes to call it. :)
 I am extracting this thought into a separate thread before I start
throwing out even more, crazier ideas.

In general, Metron is very opinionated about data flows right now.  We have
> Parser topologies that feed an Enrichment topology, which then feeds an
> Indexing topology.  We have useful bits of functionality (think Stellar
> transforms, Geo enrichment, etc) that are closely coupled with these
> topologies (aka data flows).

> When a user wants to parse heterogenous data from a single topic, that's
> not easy.  When a user wants enriched output to land in unique topics by
> sensor type, well, that's also not easy.    When a user wanted to skip
> enrichment of data sources, we actually re-architected the data flow to add
> the Indexing topology.

> In an ideal world, a user should be responsible for defining the data
> flow, not Metron.  Metron should provide the "useful bits of functionality"
> that a user can "plugin" wherever they like.  Metron itself should not care
> how the data is moving or what step in the process it is at.

Nick Allen <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message