metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <>
Subject [GitHub] incubator-metron pull request #509: METRON-822 Improve Fastcapa Performance
Date Tue, 04 Apr 2017 20:40:25 GMT
GitHub user nickwallen opened a pull request:

    METRON-822 Improve Fastcapa Performance

    This PR contains significant improvements to the performance and scalability of Fastcapa.
    * Previously the 'distributor' framework was used.  This did not scale well and has been
replaced with a burst-oriented design.
    * Receive and transmission functions have been separated to allow each to scale independently.
    * Additional parameters have been added to allow the process to be tuned easily.
    * Output provides basic transparency into the current state of processing.
    * If the probe is overwhelmed with more packets than can be handled, it will continue
processing the packets that it can without crashing.  
    * The fact that packets are being dropped is very clear.
    * A great deal of documentation has been added to the README.
    This change has been tested on Cisco UCS hardware with a 10G Cisco VNIC.  The probe was
able to capture 1 gbps before packets started to drop.  Additional performance tuning would
push this ceiling much higher, but for my purposes, I just needed to reach 1 gbps.  Additional
work will proceed in the future to find its true performance ceiling.
    To test the change yourself, simply spin-up the virtualized test environment which will
deploy and validate that Fastcapa can land packets in Kafka correctly.
    cd metron-deployment/vagrant/fastcapa-test-environment
    vagrant up
    ## Pull Request Checklist
    ### For all changes:
    - [ ] Is there a JIRA ticket associated with this PR?
    - [ ] Does your PR title start with METRON-XXXX?
    - [ ] Has your PR been rebased against the latest commit within the target branch?
    ### For code changes:
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the
root incubating-metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way
that is compatible for inclusion under [ASF 2.0](

    - [ ] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?

You can merge this pull request into a Git repository by running:

    $ git pull METRON-822

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #509
commit 34614cc9006a77826ed729f2c8a40f99bd84d4b4
Author: dlyle65535 <>
Date:   2017-04-04T12:15:42Z

    METRON-196 Deployment Fails Without Ansible closes apache/incubator-metron#499

commit fccb67aecca199d7aebd14b16debff4bf16f8acd
Author: KunalAggarwal <>
Date:   2017-04-04T16:14:23Z

    METRON-808: Amazon EC2 deployment fails at Expanding Volume Step (KunalAggarwal via mmiklavc)
closes apache/incubator-metron#496

commit 6410229e6a6a031f470d01ced2eb316d43bae29a
Author: Nick Allen <>
Date:   2017-03-21T17:51:55Z

    METRON-822 Improve Fastcapa Performance


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message