metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From justinleet <...@git.apache.org>
Subject [GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...
Date Wed, 06 Sep 2017 15:00:52 GMT
GitHub user justinleet opened a pull request:

    https://github.com/apache/metron/pull/734

    METRON-1158: Build backend for grouping alerts into meta alerts

    ## Contributor Comments
    The motivation for this is also in the ticket, but the idea is that we want to be able
to group alerts into a meta alert that lives and is queryable alongside the alerts from the
standard sources.  These would be manually created, particularly from the UI, as a result
of investigations and slicing and dicing done there.
    
    A REST API endpoint allows for the creation of a meta alert and another allows for retrieving
all meta alerts associated with a given alert (by GUID).  The first is obviously directly
useful, and the second is more useful for testing and validation, but may be useful in other
contexts (e.g. finding events that tend to get swept up into sets of investigations).
    
    The various search endpoints should work as expected, and be able to query on the metaalerts
index.
    
    A test plan is incoming, and a couple more things need to be checked out a bit more thoroughly,
but it should be pretty set for review.  If anyone is interested, it should be easy enough
to spin up, use the REST APIs (both the couple small meta alert ones along with the other
search ones) and interrogation of ES to see what's going on. It's not completely transparent,
because searches against the child alerts of meta alerts need to be done against the `alert.<field>`,
e.g. ` "query": "alert.guid:35adafd5-ffd1-4b80-806b-91d336b87220 OR guid:35adafd5-ffd1-4b80-806b-91d336b87220"`
    
    The basic test plan outline is going to be along the lines:
    Spin up full dev
    Create a couple meta alerts through the REST API.
    Ensure they come back as expected from queries.
    Ensure they link up appropriately as seen from the other endpoint.
    Ensure the other endpoints still work as expected (e.g. search should still work, etc.)
    
    Documentation also needs some updating and will be added (but I'll obviously take into
account anything anybody wants as I write more of it).
    
    Notes
    * Given that these are manually created, a single index is used, rather than a timed index
like for the standard alerts
    * Creation of meta alerts takes a map from GUID -> Index.  This is so we can do a multiget
on the items, rather than trying to search through every index for them.
    * A list of groups is provided as metadata for the UI to be able to reproduce whatever
slicing and dicing was used to group these alerts into a metaalert.  This could be extended
to take arbitrary metadata, but that seems like followon.  It's possible we may want to do
something akin to https://github.com/apache/metron/pull/702, but I'm not familiar enough with
it to say what the benefits of any synergy are.
    * Unfortunately, sorting has to be done on a common field. This means the metadata index
is required, at index time, to populate the common field.  In our case, we use the standard
threat score field. By default, the meta alerts will use the sum of the child alerts threat
scores, but this can also be set via configuration to "median", "min", "max", "average", and
"count".  All values are calculated and stored in the event that they have other interest,
but can't comingle with standard alerts.
      * It's possible that scripted sorting could be used to get around that limitation, but
that's a potentially much larger change that interferes with existing sorting and requires
appropriate construction of the script to handle.  Plus, scripting generally isn't enabled
by default for security reasons, iirc.
    * The templates have an "alert" nested field added to them. To my incredible annoyance,
this version of elasticsearch will fail on any searches where "alert" is missing rather than
just consider it not found. Even though it's in a "should" clause.  This is improved on in
5.x, where there's a param to avoid failure.  The fields should be removed again as part of
that upgrade.
    
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines
and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).

    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically
master)?
    
    
    ### For code changes:
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested
manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the
root metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way
that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?

    - [ ] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered
by building and verifying the site-book? If not then run the following commands and the verify
changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal
repository such that your branches are built there before submitting a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/metron meta_backend

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/734.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #734
    
----
commit 09c009d3530ead5d5ef7ac7d1ac4e57f39592658
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-08-03T11:59:01Z

    Meta Alerts WIP

commit 274a49f453cdccfce5251eb78aa2c51216b5faf2
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-08-30T01:47:32Z

    Trying to fix integration test

commit 0bdf67a3103a4a44b206c6ae4147738c807ba7ba
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-08-30T11:21:29Z

    test fixes

commit 8fa96167a271551e29f29a4d2e427e87151bf2a6
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-08-30T11:47:36Z

    Handling meta implementation in config

commit 4806f6ab0905b093b0e97b7a7892903a4dac4319
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T01:27:07Z

    Removing extraneous notes file and adding README section

commit 01cc5009304a653e39e32495ab32426c01193018
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T02:05:10Z

    Misc cleanup

commit 4705fe83790dcae80ea0d6270f1d2af2d2e0e4ac
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T04:52:50Z

    Fixes

commit ad4e76d91f18c1181abe3ad741fe52ae252ce68d
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T05:28:33Z

    Updating templates to resolve a problem I foresaw then forgot.  Also passing through another
function in meta dao

commit 5aec2fbae9034227bd0ea7c38f6d4c479dd0283a
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T12:49:49Z

    Adjusting sorting

commit 5d2ef582681ef3c5e2d041c2d7c2e6dc47ec47e3
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-09-06T13:12:00Z

    Updating mapping and index config

----


---

Mime
View raw message