metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-35) Implement threat intelligence message enrichment
Date Sat, 13 Feb 2016 06:46:18 GMT

    [ https://issues.apache.org/jira/browse/METRON-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145840#comment-15145840
] 

ASF GitHub Bot commented on METRON-35:
--------------------------------------

GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/22

    METRON-35 Implement threat intelligence message enrichment

    Create the infrastructure to
    * Bulk ingest threat intelligence feeds from CSV and Stix data sources into HBase
    * Enrich messages who have fields which match the threat intelligence data in HBase
    * Create the infrastructure to remove unused threat intelligence data
    * Augment the Packet capture topology to incorporate a malicious IP threat intel tagger
    
    The tagging infrastructure much meet the following criteria:
    * They are downstream of the enrichments
    * The threat intelligence bolts execute in parallel with a similar architecture as the
enrichments (i.e. split and join).


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron Threat_Intel_Feeds

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/22.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22
    
----
commit 5cf5409472d9557f7725ad14a8bcca3663c364aa
Author: cstella <cestella@gmail.com>
Date:   2016-02-03T21:30:13Z

    Added ThreatIntelBulkLoader

commit 77105eb645dd357d512aa1d52e9d28e3641003f3
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:00:16Z

    updating threat intel loader.

commit 4fcaebcdc38cbf56df89137883c92725e80a88e6
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:40:44Z

    Adding shell script to execute the threat intel feeds.

commit 0d390fc0d86af24976649828a8853aec10ab9b0c
Author: cstella <cestella@gmail.com>
Date:   2016-02-03T21:30:13Z

    Added ThreatIntelBulkLoader

commit 8256e22f679896c18df8cbfc2dd0bc67a7718b32
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:00:16Z

    updating threat intel loader.

commit e5aeb99fb29da3d00eabe53252d88a3345d5e34a
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:40:44Z

    Adding shell script to execute the threat intel feeds.

commit cfcd709bbbef3e24a5c75b41d07beae9934fe843
Author: cstella <cestella@gmail.com>
Date:   2016-02-04T16:52:37Z

    Merge branch 'Threat_Intel_Feeds' of github.com:cestella/incubator-metron into Threat_Intel_Feeds

commit 5ca646a94f91ec6745abda8fe27a585f1a15904e
Author: cstella <cestella@gmail.com>
Date:   2016-02-05T22:31:11Z

    Moving around some components to common, refactoring some dependencies to allow hbase
integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom
filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used
keys.

commit b7721d375c79e0380d0799ad895faa8b44546e76
Author: cstella <cestella@gmail.com>
Date:   2016-02-05T22:31:22Z

    Moving around some components to common, refactoring some dependencies to allow hbase
integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom
filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used
keys.

commit 6e026600e41e766a4af0e8c0caa0dc2c882d0bd9
Author: cstella <cestella@gmail.com>
Date:   2016-02-08T18:37:15Z

    Adding uni ttests for the bulk load/delete jobs.

commit 32b198cd241a296f0f1c90cbcdbdb2bcaa3e9dd6
Author: cstella <cestella@gmail.com>
Date:   2016-02-08T19:17:40Z

    Merge branch 'master' into Threat_Intel_Feeds

commit 5c0283c09217f29863ec75c49fd32b420d4e970c
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T17:52:02Z

    Updating to add new extractor, Stix extractor

commit 110ed867a0ba7ed638fab7eeb99ffe5e03dcb17e
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T18:05:51Z

    Added test for stix extractor.

commit 3cc67d58c08ef8b7cbe2d360512bdfa968e2888e
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T20:01:49Z

    Changed the bloom filter persistent access tracker to use HBase instead of HDFS

commit d49496dcb34208fdf997c01a50379ef297a9f3e4
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T20:21:58Z

    Updating poms to allow more memory.

commit c46b4c5b2cd816e50bda050fa51c0e6b28fcf3c2
Author: cstella <cestella@gmail.com>
Date:   2016-02-09T23:15:51Z

    we really need to stop shipping hbase-site.xmls around.

commit 920223ab2c39e834fddea18353997111d8693488
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:18:49Z

    Made HBase Bolt more adaptable.

commit 580257e27b917bd029eecab49a3b6b8aac375fde
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:27:00Z

    Merge branch 'master' into Threat_Intel_Feeds

commit 560877b6c29903fd80b23cb846176dca801336dc
Author: cstella <cestella@gmail.com>
Date:   2016-02-10T20:50:51Z

    HBaseBolt was so wrong.

commit 5221eb9d9f4bef6cf580efbb6a3a6848cbeda45c
Author: cstella <cestella@gmail.com>
Date:   2016-02-11T14:46:13Z

    Adding a ThreatIntelAdapter to the EnrichmentSplitterBolt

commit 716cd1ebf799b3813a2bb30c62d740945f3d93bd
Author: cstella <cestella@gmail.com>
Date:   2016-02-12T04:43:33Z

    Finalizing topologies.

commit ffb437ce6023a65473e6e49a295b45cf6df84b3d
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T01:06:42Z

    Adding vagrant setup and correcting dependency issues related to guava.

commit 6b074e02cfcb605a59f9ad7d871e5d71f2546ee8
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T04:18:21Z

    Fixed issues with dependencies and remote topology for pcap

commit db5652a0774cc51cd0ffdd62d54631d1cd2e8578
Author: cstella <cestella@gmail.com>
Date:   2016-02-13T06:36:16Z

    Fixed pom to do shading in the proper order.

----


> Implement threat intelligence message enrichment
> ------------------------------------------------
>
>                 Key: METRON-35
>                 URL: https://issues.apache.org/jira/browse/METRON-35
>             Project: Metron
>          Issue Type: New Feature
>            Reporter: Casey Stella
>            Assignee: Casey Stella
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Create the infrastructure to 
> * Bulk ingest threat intelligence feeds from CSV and Stix data sources into HBase
> * Enrich messages who have fields which match the threat intelligence data in HBase
> * Create the infrastructure to remove unused threat intelligence data
> * Augment the Packet capture topology to incorporate a malicious IP threat intel tagger
> The tagging infrastructure much meet the following criteria:
> * They are downstream of the enrichments
> * The threat intelligence bolts execute in parallel with a similar architecture as the
enrichments (i.e. split and join).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message