metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] metron pull request #733: Metron 1156
Date Tue, 05 Sep 2017 21:47:33 GMT
GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/733

    Metron 1156

    
    Troubleshooting issues when programming against a live stream of data is difficult. It
would be useful to have a means to run the entire threat triage process within the REPL before
deploying your rule set on a Metron cluster.  This creates a set of functions to allow simulation
of Threat Triage inside of the Stellar REPL.  This is useful for creating new triage rules,
debugging existing triage rules, and to iterate quickly when testing rule sets.
    
    ### Changes
    
    Created the following new functions.
       * `THREAT_TRIAGE_INIT`
       * `THREAT_TRIAGE_SCORE`
       * `THREAT_TRIAGE_CONFIG`
    
    Modified these functions to preserve existing functionality, along with enhancing them
to work with the newly created functions.
       * `THREAT_TRIAGE_PRINT`
       * `THREAT_TRIAGE_ADD`
       * `THREAT_TRIAGE_REMOVE`
       * `THREAT_TRIAGE_SET_AGGREGATOR`
    
    ### Testing
    
    The following shows how the new functions can be used.
    
    1. Create a threat triage engine.
    
        ```
        [Stellar]>>> t := THREAT_TRIAGE_INIT()
        [Stellar]>>> t
        ThreatTriage{0 rule(s)}
        ```
        
    1. Add a few triage rules.
    
        ```
        [Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule1", "rule":"value>10",
"score":10})
        {
          "enrichment" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { }
          },
          "threatIntel" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { },
            "triageConfig" : {
              "riskLevelRules" : [ {
                "name" : "rule1",
                "rule" : "value>10",
                "score" : 10.0
              } ],
              "aggregator" : "MAX",
              "aggregationConfig" : { }
            }
          },
          "configuration" : { }
        }
        ```
        ```
        [Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule2", "rule":"value>20",
"score":20})
        {
          "enrichment" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { }
          },
          "threatIntel" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { },
            "triageConfig" : {
              "riskLevelRules" : [ {
                "name" : "rule1",
                "rule" : "value>10",
                "score" : 10.0
              }, {
                "name" : "rule2",
                "rule" : "value>20",
                "score" : 20.0
              } ],
              "aggregator" : "MAX",
              "aggregationConfig" : { }
            }
          },
          "configuration" : { }
        }
        ```
        ```
        [Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule3", "rule":"value>30",
"score":30})
        {
          "enrichment" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { }
          },
          "threatIntel" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { },
            "triageConfig" : {
              "riskLevelRules" : [ {
                "name" : "rule1",
                "rule" : "value>10",
                "score" : 10.0
              }, {
                "name" : "rule2",
                "rule" : "value>20",
                "score" : 20.0
              }, {
                "name" : "rule3",
                "rule" : "value>30",
                "score" : 30.0
              } ],
              "aggregator" : "MAX",
              "aggregationConfig" : { }
            }
          },
          "configuration" : { }
        }
        ```
    
    1. Review the rules that you have created.
        ```
        [Stellar]>>> THREAT_TRIAGE_PRINT(t)
        ╔═══════╤═════════╤═════════════╤═══════╤════════╗
        ║ Name  │ Comment │ Triage Rule │ Score │ Reason ║
        ╠═══════╪═════════╪═════════════╪═══════╪════════╣
        ║ rule1 │         │ value>10    │ 10    │        ║
        ╟───────┼─────────┼─────────────┼───────┼────────╢
        ║ rule2 │         │ value>20    │ 20    │        ║
        ╟───────┼─────────┼─────────────┼───────┼────────╢
        ║ rule3 │         │ value>30    │ 30    │        ║
        ╚═══════╧═════════╧═════════════╧═══════╧════════╝
        ```
    
    1. Create a few test messages to simulate your telemetry.
        ```
        [Stellar]>>> msg1 := "{ \"value\":22 }"
        [Stellar]>>> msg1
        { "value":22 }
        ```
        ```
        [Stellar]>>> msg2 := "{ \"value\":44 }"
        [Stellar]>>> msg2
        { "value":44 }
        ```
    
    1. Score a message based on the rules that have been defined.  The result allows you to
see the total score, the aggregator, along with details about each rule that fired.
    
        ```
        [Stellar]>>> THREAT_TRIAGE_SCORE( msg1, t)
        {score=20.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0,
name=rule2, rule=value>20}]}
        ```
        ```
        [Stellar]>>> THREAT_TRIAGE_SCORE( msg2, t)
        {score=30.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0,
name=rule2, rule=value>20}, {score=30.0, name=rule3, rule=value>30}]}
        ```
    
    1. Now that I know the rule set has worked well, I can extract the configuration and push
it into my live, Metron cluster.
    
        ```
        [Stellar]>>> conf := THREAT_TRIAGE_CONFIG( t)
        [Stellar]>>> conf
        {
          "enrichment" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { }
          },
          "threatIntel" : {
            "fieldMap" : { },
            "fieldToTypeMap" : { },
            "config" : { },
            "triageConfig" : {
              "riskLevelRules" : [ {
                "name" : "rule1",
                "rule" : "value>10",
                "score" : 10.0
              }, {
                "name" : "rule2",
                "rule" : "value>20",
                "score" : 20.0
              }, {
                "name" : "rule3",
                "rule" : "value>30",
                "score" : 30.0
              }],
              "aggregator" : "MAX",
              "aggregationConfig" : { }
            }
          },
          "configuration" : { }
        }
        ```
        ```
        [Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "bro")
        ```


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1156

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/733.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #733
    
----
commit 5ed6bcbe9c312f0132f21bec44cc109b1e9fe111
Author: Nick Allen <nick@nickallen.org>
Date:   2017-09-05T02:29:41Z

    METRON-1156 Simulate Triage Rules in the Stellar REPL

commit 3fb4accc9175a2920aa6aa9ebf0d6f96e4d30c7c
Author: Nick Allen <nick@nickallen.org>
Date:   2017-09-05T21:46:03Z

    Removed unnecessary import

----


---

Mime
View raw message