metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Enrich enrichment
Date Mon, 09 Jan 2017 16:10:42 GMT
You could do that, but at the moment, I believe the indexing topology is
looking for its configs in the same place in zookeeper, so that'd get in
the way of that.

On Mon, Jan 9, 2017 at 11:08 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> What about having multiple instances of the enrichment topology with
> different configurations?  Then you would just have the last one terminate
> to indexing?  Or something like that.
>
>
> On January 9, 2017 at 10:56:38, Casey Stella (cestella@gmail.com) wrote:
>
> I think that would be a good feature to add to have arbitrary number of
> phases, though it might be tricky to code (the way I envisioned it would
> involve a loop in storm, which is possible[1]), might have unintended
> consequences to guarantees (e.g. updating enrichments might not be able to
> be applied in realtime) and could be tricky to reason about
> performance-wise.
>
> As it stands, the number of phases is a consequence of the topology
> itself.  We do not currently have an architecture which would allow an
> arbitrary number of phases without changing the flux file itself.  What you
> can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> on previous enrichments) because it's just a list of stellar statements.
> The consequence, of course, is that these statements get run within the
> same worker, which is unfortunate, but may be a stopgap workaround.
>
> *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
>
> On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
>> Maybe the naming of the phases is misleading?  What if you could set up
>> an arbitrary number of stages, with defaults?
>>
>>
>> On January 8, 2017 at 16:25:01, Casey Stella (cestella@gmail.com) wrote:
>>
>> You could do the geo enrichment normally and do a stellar hbase enrichment
>> in the threat Intel phase.
>>
>> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman <merrimanr@gmail.com> wrote:
>>
>> > Hbase enrichments and geo enrichments are done in parallel so I would
>> not
>> > expect this to work. You could do the Hbase enrichment as a threat Intel
>> > enrichment and that should work because enrichments and threat Intel are
>> > done in series.
>> >
>> >
>> >
>> > The ideal way would be to chain together Stellar enrichments but I don't
>> > think there is a geo enrichment function created yet. I think that
>> should
>> > be a Jira. I know someone is working on an update to how we do geo
>> > enrichments so I will file a follow on Jira if it's not included in the
>> > scope of that work.
>> >
>> >
>> >
>> > Ryan
>> >
>> >
>> >
>> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov <Dima.Kovalyov@sstech.us>
>> > wrote:
>> >
>> > >
>> >
>> > > Is it possible to enrich enrichment?
>> >
>> > >
>> >
>> > > For example I have IP address, I enrich it with geo and get City name,
>> >
>> > > now I want to enrich City name with city crime level (assume I have
>> that
>> >
>> > > data). But when I do that it just does not work. I specify enrichment
>> >
>> > > like that:
>> >
>> > >> {
>> >
>> > >> "index" : "msexchange",
>> >
>> > >> "batchSize" : 5,
>> >
>> > >> "enrichment" : {
>> >
>> > >> "fieldMap" : {
>> >
>> > >> "geo" : [ "destination_ip", "source_ip" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
>> >
>> > >> },
>> >
>> > >> "fieldToTypeMap" : {
>> >
>> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>> >
>> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>> >
>> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
>> >
>> > >> },
>> >
>> > >> "config" : { }
>> >
>> > >> },
>> >
>> > >> "threatIntel" : {
>> >
>> > >> "fieldMap" : { },
>> >
>> > >> "fieldToTypeMap" : { },
>> >
>> > >> "config" : { },
>> >
>> > >> "triageConfig" : {
>> >
>> > >> "riskLevelRules" : { },
>> >
>> > >> "aggregator" : "MAX",
>> >
>> > >> "aggregationConfig" : { }
>> >
>> > >> }
>> >
>> > >> },
>> >
>> > >> "configuration" : { }
>> >
>> > >> }
>> >
>> > > I tried all the ways how enrichment field can be entered just to be
>> sure
>> >
>> > > I do not mistype it.
>> >
>> > >
>> >
>> > > - Dima
>> >
>> >
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message