metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-400) Deploy Probes to running Docker Container
Date Mon, 05 Sep 2016 00:39:20 GMT

    [ https://issues.apache.org/jira/browse/METRON-400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463767#comment-15463767
] 

ASF GitHub Bot commented on METRON-400:
---------------------------------------

GitHub user dlyle65535 opened a pull request:

    https://github.com/apache/incubator-metron/pull/241

    METRON-400 Deploy Probes to running Docker Container

    This was tested in the following ways:
    Regression Testing ##
    [X] Full Dev - Worked as expected
    [X] EC2 - Worked as expected
    Functional Testing ##
    [X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari)
Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
    Steps to Stand-Alone Test ##
    *Note:* 
    You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
    Ansible will error without docker.py installed and give you a chance to:
    ```
    pip install 'docker-py>=1.7.0'
    ```
    Provision Container ###
    Start the container: 
    ``` 
    docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
    ```
    Run a shell on the container: 
    ```
    docker exec -it amb-server bash
    ```
    Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
    ```
    vi /etc/resolv.conf
    ```
    Get HDP repo:
    ```
    wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo
-O /etc/yum.repos.d/hdp.repo
    ```
    Install Zookeeper:
    ```
    yum install -y zookeeper-server
    ```
    Setup and Start Zookeeper:
    ```
    export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
    export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
    export ZOO_LOG_DIR=/var/log/zookeeper
    export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
    export SERVER_JVMFLAGS=-Xmx1024m
    export JAVA=$JAVA_HOME/bin/java
    export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
    export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
    export ZOOCFG=zoo.cfg
    source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
    /usr/hdp/current/zookeeper-server/bin/zkServer.sh start
    ```
    Test Zookeeper:
    ```
    /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
    ```
    The command should return:
    > Connecting to localhost:2181
    > 
    > WATCHER::
    > 
    > WatchedEvent state:SyncConnected type:None path:null
    > [zookeeper]
    >
    Install and Start Kafka:
    ``` 
    yum install -y kafka
    /usr/hdp/current/kafka-broker/bin/kafka start
    ```
    Test Kafka:
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic test --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092
--topic test
    ```
    Add some test data and hit ctrl-c.
    ```
    /usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181
--topic test --from-beginning
    ```
    You should see your test data - ctrl-c to exit.
    Create Probe Data Topics:
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic bro --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic snort --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic yaf --replication-factor 1 --partitions 1
    ```
    Exit Container Shell:
    ```
    exit
    ```
    Install Probes using Ansible:
    1. cd <metron-home>/metron-deployment/playbooks
    2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url:
amb-server:9092)
    3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
    ansible-playbook docker_probe_install.yml
    4. run ansible-playbook
    ```
    ansible-playbook docker_probe_install.yml
    ```
    It should complete with:
    > PLAY RECAP *********************************************************************
    > amb-server : ok=100 changed=60 unreachable=0 failed=0
    > localhost : ok=2 changed=0 unreachable=0 failed=0
    > docker exec -it amb-server bash
    Start Sensor Probes:
    ```
    service pcap-replay start
    /usr/local/bro/bin/broctl start
    service yaf start eth0
    service snortd start
    /usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n
snort > /dev/null 2>&1 &
    ```
    Check for Data
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic bro
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic snort
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic yaf
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #241
    
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dlyle65535@gmail.com>
Date:   2016-08-31T16:29:53Z

    METRON-400 Deploy Probes to running Docker Container

commit 26d0561748da28d3a6e562334816dd6ec04d4e46
Author: David Lyle <dlyle65535@gmail.com>
Date:   2016-09-04T17:41:03Z

    Only use amb4 for Kafka broker

----


> Deploy Probes to running Docker Container
> -----------------------------------------
>
>                 Key: METRON-400
>                 URL: https://issues.apache.org/jira/browse/METRON-400
>             Project: Metron
>          Issue Type: New Feature
>            Reporter: David M. Lyle
>            Assignee: David M. Lyle
>
> Sometimes we test using kafka containers. Create playbook to deploy to docker container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message