metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cestella <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Date Mon, 06 Feb 2017 14:47:51 GMT
GitHub user cestella reopened a pull request:

    https://github.com/apache/incubator-metron/pull/432

    METRON-682: Unify and Improve the Flat File Loader

    Currently the flat file loader is deficient in a couple ways:
    * It only supports importing local data despite there being a separate, poorly named,
application which supports importing enrichment via MapReduce called threat_intel_loader.sh
    * It does not support local imports from HDFS
    * It does not support local imports from URLs
    * It does not support importing zipped archives locally
    * You cannot import more than one file at once
    
    This JIRA will:
    * Unify the MapReduce and local imports into one program and allow the user to specify
the import mode with a CLI flag
    * Support local imports from HDFS and URLs
    * Support local imports from zipped files
    * Support importing more than one file at once
    
    Testing plan in comments

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron unified_loader

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #432
    
----
commit 47d814ef95d67738d20ce5dc530ba7b05d418a96
Author: cstella <cestella@gmail.com>
Date:   2017-01-27T23:15:44Z

    Multithreading the SimpleEnrichmentFlatFileLoader

commit 918d4ce4aea5d7dfde992f32bf049c70f35dd182
Author: cstella <cestella@gmail.com>
Date:   2017-01-27T23:23:19Z

    doc changes.

commit c6ca3a86881eb77bc9598a61e3c0cf8280ccb03f
Author: cstella <cestella@gmail.com>
Date:   2017-01-27T23:39:56Z

    Updating docs.

commit 8c9a79cdfa38ea2fbd161095d5e346147558ec5f
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T03:36:31Z

    Investigating integration tests.

commit 315bd181aa634290ab987441d81c28addb7952e2
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T04:09:28Z

    Update integration test to be a proper integration test.

commit 004c6f41b6c1cc3ecea70513e1a468501bd32e3c
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T04:49:37Z

    Adding spliterator unit test for completeness

commit f8dd48ef920c948e1fc5ff736e386f641e551b2b
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T05:01:42Z

    Updating test to use a proper file

commit 9b04f9723d442c8f4fb7a8bcaa1d733fc1305dc4
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T05:17:12Z

    Updating docs and renaming a few things.

commit eb5b82cc35bd767a169f548ea8144dd9ae165f84
Author: cstella <cestella@gmail.com>
Date:   2017-01-28T05:23:25Z

    Update one more test case.

commit 310c98bd946b2fdb320193cce85d368f016bf8c3
Author: cstella <cestella@gmail.com>
Date:   2017-01-30T20:36:23Z

    Merge branch 'master' into unified_loader

commit 3f6e3ba4f30e41c94ff25027f1fd7c839ea6c9bf
Author: cstella <cestella@gmail.com>
Date:   2017-01-31T15:39:03Z

    Updating simple enrichment flat file loader to be complete.

commit 79cfdb4fba5e82e9e170bfc77c7133e6646f9787
Author: cstella <cestella@gmail.com>
Date:   2017-01-31T22:12:05Z

    Removing old threatintel_bulk_load.sh script and integrating into the flatfile load script

commit bf7756b52e66907ca23a576ba9be9ab40b33f77d
Author: cstella <cestella@gmail.com>
Date:   2017-01-31T22:22:17Z

    Forgot licenses.

commit a104f464e6b882121c7ab44079a5570d282c8457
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T00:28:46Z

    updating script.

commit b121e13d892834865847ddd806cbf10da63fa44e
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T00:34:28Z

    Merge branch 'master' into unified_loader

commit b5a9e5a9243576b27d59e959dfab3e99d34eb761
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T00:57:02Z

    Added gzip and zip to regular files

commit 323267ddfb52ab1aa7488e02643a8158044797e2
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T15:04:53Z

    Fixed stupid zip issue.

commit bc26b5b3992b91097bb4fc4b214d4b6bacaddfbb
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T16:27:58Z

    Updating readme and making progress bar optional and better.

commit 6cdf35d94f72be7da524fd5f854876f131ddb9f9
Author: cstella <cestella@gmail.com>
Date:   2017-02-01T17:39:59Z

    updating tests to include gzip and zip

commit 799811ccd587d9fcd2135ea3970d5bbf1121a29b
Author: cstella <cestella@gmail.com>
Date:   2017-02-03T22:21:34Z

    Reacted to @mmiklavcic

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message