metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-25) Create Bro Plugin to Send Logs Directly to Kafka
Date Thu, 04 Feb 2016 21:19:39 GMT

    [ https://issues.apache.org/jira/browse/METRON-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133056#comment-15133056
] 

ASF GitHub Bot commented on METRON-25:
--------------------------------------

Github user nickwallen commented on the pull request:

    https://github.com/apache/incubator-metron/pull/17#issuecomment-180056871
  
    The values in `bro-plugin-kafka/scripts/init.bro` are merely defaults.  They could even
be completely removed from there.  I just find them useful so that a user doesn't have to
define all of the configuration values all of the time.  For example a user will rarely want
to change `max_wait_on_delivery`.
    
    The way to configure the kafka broker and topic name as a user of this plugin is described
in the README.  You define these in your `.../site/local.bro` script so that it looks something
like the following:
    
    ```
    @load Metron/Kafka/logs-to-kafka.bro
    redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
    redef Kafka::kafka_broker_list = "localhost:9092";
    redef Kafka::topic_name = "bro";
    ```
    
    As an example, you can see how the Ansible scripts configure these values in `deployment/roles/bro/tasks/bro-plugin-kafka.yml`.
    
    ```
    - name: Configure bro plugin
      lineinfile:
        dest: /usr/local/bro/share/bro/site/local.bro
        line: "{{ item }}"
      with_items:
        - "@load Metron/Kafka/logs-to-kafka.bro"
        - "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);"
        - "redef Kafka::kafka_broker_list = \"{{ kafka_broker_url }}\";"
        - "redef Kafka::topic_name = \"{{ bro_topic }}\";"
    ```
    
    Good eye.  Does that make sense?


> Create Bro Plugin to Send Logs Directly to Kafka
> ------------------------------------------------
>
>                 Key: METRON-25
>                 URL: https://issues.apache.org/jira/browse/METRON-25
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Nick Allen
>            Priority: Critical
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Create a Bro plugin that will consume the logs produced by Bro and send them directly
to a Kafka topic.  The types of logs to send should be configurable, so that only a subset
of them are published to Kafka.  For example, I may only want DNS::LOG and HTTP::LOG sent
to Kafka.  This should not interfere with the existing file based logging which is useful
for diagnostics and troubleshooting.
> The alternative solution to creating this Bro plugin is to use some means of tailing
the log files that are generated by Bro. Each stream in Bro is logged to a separate file,
so you'd have to tail each of these files independently. Tailing log files like this is problematic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message