metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JonZeolla <...@git.apache.org>
Subject [GitHub] metron issue #805: METRON-1261: Apply bro security patch
Date Wed, 18 Oct 2017 13:47:39 GMT
Github user JonZeolla commented on the issue:

    https://github.com/apache/metron/pull/805
  
    # Testing
    1.  Create a working directory and pull in this PR
        ```
        mkdir ~/metron-1261
        git clone https://github.com/apache/metron ~/metron-1261/metron
        cd ~/metron-1261/metron
        git remote add jonzeolla https://github.com/jonzeolla/metron
        git pull jonzeolla METRON-1261
        ```
    1.  Modify [this](https://github.com/JonZeolla/metron/blob/METRON-1261/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
to remove `sensors,` (to spin up the real sensors).
        ```
        sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile
        ```
    1.  Set up the environment in full-dev.
        ```
        vagrant ssh
        sudo su -
        export PATH=$PATH:/usr/local/bro/bin
        service monit stop && service sensor-stubs stop bro && broctl stop
        yum -y install jq wireshark
        ```
    1.  Configure kafka in local.bro so all of the currently supported bro logs are being
sent.
        ```
        sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG,
DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG,
Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG,
X509::LOG);/' /usr/local/bro/share/bro/site/local.bro
        echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro
        echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
        echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro
        sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro
        ```
    1.  Monitor the bro kafka topic
        ```
        # Open a new terminal
        cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
        vagrant ssh
        sudo su -
        export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
        kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
        ```
    1.  Monitor the storm logs.
        ```
        # Open a new terminal
        cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
        vagrant ssh
        sudo su -
        export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
        # Look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr
are expected, and should be addressed as a part of METRON-939)
        tail -f /var/log/storm/workers-artifacts/indexing-*/*/worker.log | grep -i "org.elasticsearch.index.mapper.MapperParsingException:
failed to parse"
        # You may want to evaluate worker.log for other errors, but the prior command is helpful
to cut through some of the failed indexing of IPv6 addresses
        ```
    1.  Run bro against some public pcaps.
        ```
        # In the first of your three terminals
        # These are kept separate so that the flat file log output won't stomp the prior ones,
for ingest validation
        mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic ~/brotmp/ssh ~/brotmp/ftp ~/brotmp/radius
        wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap
        wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ~/brotmp/nitroba/nitroba.pcap
        wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap
        wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ~/brotmp/ftp/ftp.pcap
        wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ~/brotmp/radius/radius_localhost.pcapng
        cd ~/brotmp/example-traffic
        bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
        cd ~/brotmp/nitroba
        bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
        cd ~/brotmp/ssh
        bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C
        cd ~/brotmp/ftp
        bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C
        cd ~/brotmp/radius
        editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap
        bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C
        ```
    1.  Validate that terminals 2 and 3 don't have any errors that you don't expect.
    1.  Verify proper indexing in ES and availability in kibana.
        ```
        # Check around and make sure things look okay
        declare -a exists notexists; for protocol in http dns conn dpd dhcp ftp ssh ssl smtp
radius weird files notice software known_certs x509 known_devices; do if [[ $(curl -s -XGET
"node1:9200/bro*/_search?q=protocol:${protocol}" | jq '.hits.hits') == '[]' ]]; then notexists+="
${protocol}"; else exists+=" ${protocol}"; fi; done; if [ ${#notexists[@]} -ne 0 ]; then echo
-e "\n\n\033[0mThe following do exist in ES: ${exists[@]}\n\033[0;31mThe following do NOT
exist in ES: ${notexists[@]}\033[0m"; else echo 'All of the log types are in ES!  Success!';
fi; unset exists notexists
        # Check Kibana.  For example:  http://node1:5000/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((id:'3',params:(field:protocol,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=bro*&type=histogram
        # OPTIONAL testing
        # Run `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head` and look
around
        curl -XGET node1:9200/_cat/indices # First column should be all green
        curl -XGET "node1:9200/bro*/_count" # Check the count of entries in the bro index,
you can re-run bro against specific PCAPs and watch this increase, etc.
        ```


---

Mime
View raw message