metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From justinleet <>
Subject [GitHub] incubator-metron pull request #505: METRON-817: Customise output file path p...
Date Mon, 03 Apr 2017 13:26:52 GMT
GitHub user justinleet opened a pull request:

    METRON-817: Customise output file path patterns for HDFS indexing

    ## Contributor Comments
    Primarily this affects HdfsWriter by changing the output path from a set path (`/apps/metron/.../<sensor>`),
and allow it to be defined via a Stellar Function.  Specifically, the base path is still defined
the same (The `/apps/metron/.../` portion), but the `<sensor>` portion is dropped and
can now be defined by a Stellar function.  By default, the original behavior of `<sensor>`
is used.  This is defined in the `<sensor>.json` file as indicated in the new
for metron-writer.
    ### Notes
    - This requires adding tracking things a bit more carefully (and if you're reviewing,
please validate that it happens correctly).  When the outputFile is closed, we remove the
sourceHandler from HdfsWriter's map.
      - I'm slightly concerned about the correctness of the implementation, but it seems necessary
to ensure that we don't leave a bunch of SourceHandlers lying around as data changes (and
we don't want an enormous number of output files being written to).
      - If there's a cleaner way to manage this, I'd love to hear it and can refactor pretty
easily. It throws off the rotation count (because we kill the SourceHandler from the map itself),
but I doubt we care about that since it really only shows up in the output filename anyway.
    - This also adds an argument for max open files.  This is a flux level config. I defaulted
this to 500.  500 was chosen because it was an arbitrary round number that wasn't enormous.
      - If someone has a default with any real reasoning behind it, I'll go ahead and change
    - In HdfsWriter, we iterate through the messages, apply the Stellar function and then
call the relevant handler. The entire group of message is treated as one single pass/fail
(which is the same as the old behavior), rather than individually. The try/catch could potentially
be moved into the for loop, but I don't think there's an explicit link between the message
and the tuples that we can exploit to fail per message.  I don't think it needs to be addressed
here, but I'm curious if there's thought on this.
    ### Testing
    Unit tests are added to pretty much cover HdfsWriter, and this can be spun up in a dev
    To test in dev
    - Spin up a dev environment
    - Validate that the output matches the old format in HDFS (Nothing has an output function
      [hdfs@node1 vagrant]$ hdfs dfs -ls /apps/metron/indexing/indexed/
      Found 3 items
      drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/bro
      drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/error
      drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/snort
    - Edit the indexing config for Bro to include an outputPathFunction in the hdfs section,
e.g. in `/usr/metron/0.3.1/config/zookeeper/indexing/bro.json`
        "hdfs" : {
          "index": "bro",
          "batchSize": 5,
          "enabled" : true,
          "outputPathFunction": "FORMAT('ipsrc-%s', ip_src_addr)"
        "elasticsearch" : {
          "index": "bro",
          "batchSize": 5,
          "enabled" : true
        "solr" : {
          "index": "bro",
          "batchSize": 5,
          "enabled" : true
    - Push the config configs to ZooKeeper: `/usr/metron/0.3.1/bin/ -z node1:2181
-m PUSH -i /usr/metron/0.3.1/config/zookeeper/`
    - Let some more data run through and check the output folders, e.g.
    [hdfs@node1 vagrant]$ hdfs dfs -ls /apps/metron/indexing/indexed/
    Found 5 items
    drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/bro
    drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/error
    drwxrwxr-x   - storm hadoop          0 2017-04-03 13:14 /apps/metron/indexing/indexed/ipsrc-
    drwxrwxr-x   - storm hadoop          0 2017-04-03 13:14 /apps/metron/indexing/indexed/ipsrc-
    drwxrwxr-x   - storm hadoop          0 2017-04-03 13:11 /apps/metron/indexing/indexed/snort
    [hdfs@node1 vagrant]$ hdfs dfs -ls /apps/metron/indexing/indexed/ipsrc-
    Found 1 items
    -rw-r--r--   1 storm hadoop     223182 2017-04-03 13:14 /apps/metron/indexing/indexed/ipsrc-
    ## Pull Request Checklist
    Thank you for submitting a contribution to Apache Metron (Incubating).  
    Please refer to our [Development Guidelines](
for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](
for complete smoke testing guides.  
    In order to streamline the review of the contribution we ask you follow these guidelines
and ask you to double check the following:
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](

    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested
    - [x] Have you ensured that the full suite of tests and checks have been executed in the
root incubating-metron folder via:
      mvn -q clean integration-test install && build_utils/ 
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - ~If adding new dependencies to the code, are these dependencies licensed in a way that
is compatible for inclusion under [ASF 2.0](
    - [x] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered
by building and verifying the site-book? If not then run the following commands and the verify
changes via `site-book/target/site/index.html`:
      cd site-book
      mvn site:site
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible.
    It is also recommened that [travis-ci]( is set up for your personal
repository such that your branches are built there before submitting a pull request.

You can merge this pull request into a Git repository by running:

    $ git pull hdfs_path

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #505
commit e84c1393c293a809372386fdcf374f0c3dc50d9c
Author: justinjleet <>
Date:   2017-03-31T03:34:21Z

    Allowing for message guided output and adding doc

commit 9762cb6c473c6c11da70963f8de1e24f6f7b0502
Author: justinjleet <>
Date:   2017-04-03T11:41:25Z

    renamed parameters in override method for clarity, added test around SourceFileNameFormat
to ensure additions work

commit 6693740a85a5cd9d90ac4a413e2037410f927432
Author: justinjleet <>
Date:   2017-04-03T11:47:57Z

    Removing extraneous json change, and not tripping rat

commit 72348501c4f7876aaaa7930f600bf68bc98a0d61
Author: justinjleet <>
Date:   2017-04-03T12:12:10Z

    Adjusting output

commit 8b19e67c10821b12c8e09439fc7c89528372df77
Author: justinjleet <>
Date:   2017-04-03T12:19:13Z

    README adjustment

commit 10b9c52ad373e395f612a8253385d6e9783cb09e
Author: justinjleet <>
Date:   2017-04-03T12:22:22Z

    Renaming SourceFileNameFormat, cleaning up a couple minor annoyances in SourceAwareMoveAction


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message