metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] metron pull request #608: METRON-986 Enhance Fastcapa to Support Intel X520
Date Mon, 05 Jun 2017 18:40:09 GMT
GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/608

    METRON-986 Enhance Fastcapa to Support Intel X520

    ### Changes
    
    The following changes have been made to Fastcapa to improve interoperability with the
Intel X520 NIC, address bugs uncovered during performance testing, and some basic enhancements
to improve ease-of-use.
    
      * Created a separate command-line parameter for the transmit worker burst size; `-w
TX_BURST_SIZE`.  This defines the maximum number of packets dequeued from the transmit ring
and enqueued for delivery to Kafka at one time.  Previously the burst size for the receive
and transmit worker were the same.  In practice the transmit worker burst size needs to be
roughly 10x greater than the receive worker burst size.
    
      * Fixed a bug that caused some failures reported through the message delivery callback
to be ignored.  Both successful and failed deliveries are reported through this callback and
the logic has been corrected to match this.
    
      * Error messages for each failed message can be logged when the `DEBUG` flag is used
during compilation.  By default errors for each message are not logged due to performance
implications.
    
      * Now logging all configuration parameters by default.
    
      * The code was unnecessarily limiting `-x TX_RING_SIZE` by using a uint16_t; a 32-bit
value is perfectly acceptable.  Also, no need to validate the -x TX_RING_SIZE as DPDK's EAL
does this for us.  
    
      * Added separate default port configurations for the receive and transmit side.  Previously
the same configuration values were used for both.
    
      * Added logic to count, summarize and report all ethernet device errors.
    
      * Refactored logic so that all receive and transmit worker logic is contained within
`worker.[c|h]` and all NIC logic contained within `nic.[c|h]`.
    
      * Updated the fastcapa service script to allow the process to be stopped, killed, and
the logs tailed.
    
    
    ### Testing 
    
    This change has been tested on a live cluster with the Intel X520 NIC. The probe was able
to capture and land roughly 1 gbps of raw packet data into Kafka.  The change was also tested
in the Fastcapa test environment.
    
    To test the change yourself, simply spin-up the virtualized test environment which will
deploy and validate that Fastcapa can land packets in Kafka correctly.
    
    ```
    cd metron-deployment/vagrant/fastcapa-test-environment
    vagrant up
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-986

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/608.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #608
    
----
commit 5559c00b3ad569a4b6aa3a48c89bfbb21f6d4830
Author: Nick Allen <nick@nickallen.org>
Date:   2017-05-05T18:16:49Z

    METRON-986 Enhance Fastcapa to Support Intel X520

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message