metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cestella <...@git.apache.org>
Subject [GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Date Sat, 13 Feb 2016 06:45:37 GMT
GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/22

    METRON-35 Implement threat intelligence message enrichment

    Create the infrastructure to
    * Bulk ingest threat intelligence feeds from CSV and Stix data sources into HBase
    * Enrich messages who have fields which match the threat intelligence data in HBase
    * Create the infrastructure to remove unused threat intelligence data
    * Augment the Packet capture topology to incorporate a malicious IP threat intel tagger
    
    The tagging infrastructure much meet the following criteria:
    * They are downstream of the enrichments
    * The threat intelligence bolts execute in parallel with a similar architecture as the
enrichments (i.e. split and join).


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron Threat_Intel_Feeds

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/22.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22
    
----
commit 5cf5409472d9557f7725ad14a8bcca3663c364aa
Author: cstella <cestella@gmail.com>
Date:   2016-02-03T21:30:13Z

    Added ThreatIntelBulkLoader

commit 77105eb645dd357d512aa1d52e9d28e3641003f3
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:00:16Z

    updating threat intel loader.

commit 4fcaebcdc38cbf56df89137883c92725e80a88e6
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:40:44Z

    Adding shell script to execute the threat intel feeds.

commit 0d390fc0d86af24976649828a8853aec10ab9b0c
Author: cstella <cestella@gmail.com>
Date:   2016-02-03T21:30:13Z

    Added ThreatIntelBulkLoader

commit 8256e22f679896c18df8cbfc2dd0bc67a7718b32
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:00:16Z

    updating threat intel loader.

commit e5aeb99fb29da3d00eabe53252d88a3345d5e34a
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:40:44Z

    Adding shell script to execute the threat intel feeds.

commit cfcd709bbbef3e24a5c75b41d07beae9934fe843
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:52:37Z

    Merge branch 'Threat_Intel_Feeds' of github.com:cestella/incubator-metron into Threat_Intel_Feeds

commit 5ca646a94f91ec6745abda8fe27a585f1a15904e
Author: cstella <cestella@gmail.com>
Date:   2016-02-05T22:31:11Z

    Moving around some components to common, refactoring some dependencies to allow hbase
integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom
filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used
keys.

commit b7721d375c79e0380d0799ad895faa8b44546e76
Author: cstella <cestella@gmail.com>
Date:   2016-02-05T22:31:22Z

    Moving around some components to common, refactoring some dependencies to allow hbase
integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom
filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used
keys.

commit 6e026600e41e766a4af0e8c0caa0dc2c882d0bd9
Author: cstella <cestella@gmail.com>
Date:   2016-02-08T18:37:15Z

    Adding uni ttests for the bulk load/delete jobs.

commit 32b198cd241a296f0f1c90cbcdbdb2bcaa3e9dd6
Author: cstella <cestella@gmail.com>
Date:   2016-02-08T19:17:40Z

    Merge branch 'master' into Threat_Intel_Feeds

commit 5c0283c09217f29863ec75c49fd32b420d4e970c
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T17:52:02Z

    Updating to add new extractor, Stix extractor

commit 110ed867a0ba7ed638fab7eeb99ffe5e03dcb17e
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T18:05:51Z

    Added test for stix extractor.

commit 3cc67d58c08ef8b7cbe2d360512bdfa968e2888e
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T20:01:49Z

    Changed the bloom filter persistent access tracker to use HBase instead of HDFS

commit d49496dcb34208fdf997c01a50379ef297a9f3e4
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T20:21:58Z

    Updating poms to allow more memory.

commit c46b4c5b2cd816e50bda050fa51c0e6b28fcf3c2
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T23:15:51Z

    we really need to stop shipping hbase-site.xmls around.

commit 920223ab2c39e834fddea18353997111d8693488
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:18:49Z

    Made HBase Bolt more adaptable.

commit 580257e27b917bd029eecab49a3b6b8aac375fde
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:27:00Z

    Merge branch 'master' into Threat_Intel_Feeds

commit 560877b6c29903fd80b23cb846176dca801336dc
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:50:51Z

    HBaseBolt was so wrong.

commit 5221eb9d9f4bef6cf580efbb6a3a6848cbeda45c
Author: cstella <cestella@gmail.com>
Date:   2016-02-11T14:46:13Z

    Adding a ThreatIntelAdapter to the EnrichmentSplitterBolt

commit 716cd1ebf799b3813a2bb30c62d740945f3d93bd
Author: cstella <cestella@gmail.com>
Date:   2016-02-12T04:43:33Z

    Finalizing topologies.

commit ffb437ce6023a65473e6e49a295b45cf6df84b3d
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T01:06:42Z

    Adding vagrant setup and correcting dependency issues related to guava.

commit 6b074e02cfcb605a59f9ad7d871e5d71f2546ee8
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T04:18:21Z

    Fixed issues with dependencies and remote topology for pcap

commit db5652a0774cc51cd0ffdd62d54631d1cd2e8578
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T06:36:16Z

    Fixed pom to do shading in the proper order.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message