metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cestella <...@git.apache.org>
Subject [GitHub] metron issue #724: METRON-1142: Add Geo Hashing functions to stellar
Date Tue, 05 Sep 2017 19:26:17 GMT
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/724
  
    Ok, I tested this via a custom enrichment and custom parser field transformation and ensured
that no adverse classpath issues happened.  I'm ready to call this ready for review.
    
    Manual testing should follow the squid example:
    * Add a new parser field transformation:
    ```
    {
      "parserClassName": "org.apache.metron.parsers.GrokParser",
      "sensorTopic": "squid",
      "parserConfig": {
        "grokPath": "/patterns/squid",
        "patternLabel": "SQUID_DELIMITED",
        "timestampField": "timestamp"
      },
      "fieldTransformations" : [
        {
          "transformation" : "STELLAR"
        ,"output" : [ "full_hostname", "domain_without_subdomains", "geo_hash_parser"]
        ,"config" : {
          "full_hostname" : "URL_TO_HOST(url)"
          ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
          ,"geo_hash_parser" : "GEOHASH_FROM_LOC(GEO_GET(ip_dst_addr))"
                    }
        }
                               ]
    }
    ```
    * Add a new enrichment config:
    ```
    {
      "enrichment" : {
        "fieldMap": {
          "stellar": {
           "config" : [
             "geo_hash_enrichment := GEOHASH_FROM_LOC(GEO_GET(ip_dst_addr))"
           ]
          }
        }
      },
      "threatIntel": { }
    }
    ```
    * Send squid data through and ensure that `geo_hash_enrichment` and `geo_hash_parser`
yield a hash
    * Test the various new stellar commands in the REPL (e.g. `GEOHASH_FROM_LOC(GEO_GET('172.168.2.3'))`


---

Mime
View raw message