metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlyle65535 <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #241: METRON-400 Deploy Probes to running Dock...
Date Mon, 05 Sep 2016 00:38:36 GMT
GitHub user dlyle65535 opened a pull request:

    https://github.com/apache/incubator-metron/pull/241

    METRON-400 Deploy Probes to running Docker Container

    This was tested in the following ways:
    Regression Testing ##
    [X] Full Dev - Worked as expected
    [X] EC2 - Worked as expected
    Functional Testing ##
    [X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari)
Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
    Steps to Stand-Alone Test ##
    *Note:* 
    You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
    Ansible will error without docker.py installed and give you a chance to:
    ```
    pip install 'docker-py>=1.7.0'
    ```
    Provision Container ###
    Start the container: 
    ``` 
    docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
    ```
    Run a shell on the container: 
    ```
    docker exec -it amb-server bash
    ```
    Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
    ```
    vi /etc/resolv.conf
    ```
    Get HDP repo:
    ```
    wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo
-O /etc/yum.repos.d/hdp.repo
    ```
    Install Zookeeper:
    ```
    yum install -y zookeeper-server
    ```
    Setup and Start Zookeeper:
    ```
    export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
    export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
    export ZOO_LOG_DIR=/var/log/zookeeper
    export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
    export SERVER_JVMFLAGS=-Xmx1024m
    export JAVA=$JAVA_HOME/bin/java
    export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
    export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
    export ZOOCFG=zoo.cfg
    source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
    /usr/hdp/current/zookeeper-server/bin/zkServer.sh start
    ```
    Test Zookeeper:
    ```
    /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
    ```
    The command should return:
    > Connecting to localhost:2181
    > 
    > WATCHER::
    > 
    > WatchedEvent state:SyncConnected type:None path:null
    > [zookeeper]
    >
    Install and Start Kafka:
    ``` 
    yum install -y kafka
    /usr/hdp/current/kafka-broker/bin/kafka start
    ```
    Test Kafka:
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic test --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092
--topic test
    ```
    Add some test data and hit ctrl-c.
    ```
    /usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181
--topic test --from-beginning
    ```
    You should see your test data - ctrl-c to exit.
    Create Probe Data Topics:
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic bro --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic snort --replication-factor 1 --partitions 1
    /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create
--topic yaf --replication-factor 1 --partitions 1
    ```
    Exit Container Shell:
    ```
    exit
    ```
    Install Probes using Ansible:
    1. cd <metron-home>/metron-deployment/playbooks
    2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url:
amb-server:9092)
    3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
    ansible-playbook docker_probe_install.yml
    4. run ansible-playbook
    ```
    ansible-playbook docker_probe_install.yml
    ```
    It should complete with:
    > PLAY RECAP *********************************************************************
    > amb-server : ok=100 changed=60 unreachable=0 failed=0
    > localhost : ok=2 changed=0 unreachable=0 failed=0
    > docker exec -it amb-server bash
    Start Sensor Probes:
    ```
    service pcap-replay start
    /usr/local/bro/bin/broctl start
    service yaf start eth0
    service snortd start
    /usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n
snort > /dev/null 2>&1 &
    ```
    Check for Data
    ```
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic bro
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic snort
    /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181
--from-beginning --topic yaf
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #241
    
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dlyle65535@gmail.com>
Date:   2016-08-31T16:29:53Z

    METRON-400 Deploy Probes to running Docker Container

commit 26d0561748da28d3a6e562334816dd6ec04d4e46
Author: David Lyle <dlyle65535@gmail.com>
Date:   2016-09-04T17:41:03Z

    Only use amb4 for Kafka broker

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message