metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Allen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
Date Tue, 04 Sep 2018 17:14:00 GMT

     [ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nick Allen updated METRON-1533:
-------------------------------
    Fix Version/s:     (was: Next + 1)
                   0.6.0

> Create KAFKA_FIND Stellar Function
> ----------------------------------
>
>                 Key: METRON-1533
>                 URL: https://issues.apache.org/jira/browse/METRON-1533
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>            Priority: Minor
>             Fix For: 0.6.0
>
>
> When creating enrichments, I often find that I want to validate that the enrichment I
just created was successful on the live, incoming stream of telemetry. My workflow looks something
like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512,
locID=5368361, longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
>   "enrichment" : {
>     "fieldMap": {
>       "stellar": {
>         "config": [
>            "geo := GEO_GET(ip_src_addr)"
>         ]
>       }
>     }
>   },
>   "threatIntel": {
>   }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>  
>  3.  It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message.
You would just have to get lucky and hope that the enrichment worked and secondly that you
would pull down a Bro message (as opposed to a different sensor).
>  
> I would rather have a function that lets me only pull back the messages that I care about.
In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message