metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mmiklavc <>
Subject [GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...
Date Thu, 23 Mar 2017 19:44:59 GMT
Github user mmiklavc commented on the issue:
    I just finished running some parser tests in a Kerberized environment to confirm that
this will address both kerberized and non-kerberized configurations. Here is what I had to
do to get this running. Note, I'm using Ambari to configure everything for Hadoop, but not
the Metron components. There is separate work that will need to be done for Metron in the
MPack for Ambari.
    1. Spin up full dev as normal (ensure that the profile is correctly set to -P HDP-
in metron-deployment/roles/metron-builder/tasks/main.yml). As of this writing, the current
full-dev ansible install will do a complete build as part of the deployment.
    2. service stop monit and kill all the topologies. You can go ahead and kill the other
sensors per the commands from @cestella  as well (bro, pycapa, etc.). We'll spin up our own
simple test topology and populate the kafka topic manually to test this out.
    3. Setup symlinks
    ln -s /usr/metron/0.3.1/lib/metron-*-uber.jar /usr/hdp/
    ln -s /usr/hdp/current/hadoop-client/conf/core-site.xml /etc/storm/conf
    ln -s /usr/hdp/current/hbase-client/conf/hbase-site.xml /etc/storm/conf
    ln -s /usr/hdp/ /usr/hdp/
    ln -s /usr/hdp/ /usr/hdp/
    4. Check that the jce security is setup in /usr/jdk64/jdk1.8.0_77/jre/lib/security/. If
not, then you'll want to run through the following steps:
    4.1 Download the jce policy from:
and copy it to your vm.
    4.2 unzip -o -j -q /path-to-the-copied-zip/ -d /usr/jdk64/jdk1.8.0_77/jre/lib/security/
    5. Configure MIT Kerberos
    yum -y install krb5-server krb5-libs krb5-workstation
    sed -i 's/' /etc/krb5.conf
    cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
    kdb5_util create -s
    /etc/rc.d/init.d/krb5kdc start
    /etc/rc.d/init.d/kadmin start
    chkconfig krb5kdc on
    chkconfig kadmin on

    6. Setup admin and personal user principals. You'll kinit as the "{your user name}" user.
For my tests, I did "addprinc mike". I just made the passwords "password" for the sake of
easy testing.
    kadmin.local -q "addprinc admin/admin"
    kadmin.local -q "addprinc {your user name}"
    7. In Ambari, setup AutoHDFS by adding the following properties to custom-storm-site
    8. Kerberize the cluster via Ambari. The wizard is fairly straightforward, but further
documentation can be found [here](
For this exercise, choose existing MIT KDC (this is what we setup and installed in previous
steps.) Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when
testing the KDC.
    9. Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're
going to run the parsers manually against the rest of the Hadoop cluster Kerberized.
    10. Kinit and provide the password you chose from earlier.
    kinit <your username>
    11. Create a kafka topic for the jsonMap parser.
    /usr/hdp/current/kafka-broker/bin/ --zookeeper node1:2181 --create --topic
jsonMap --replication-factor 1 --partitions 1
    12. Setup the ACLs for the topic AND the consumer group. The consumer group config is
new in this API as far as I can tell because we didn't have to do this in the "storm-kafka"
version of the api. Note that the docs around kafka 0.10 will suggest using a "--consumer-group"
option, but in HDP 2.5.x the option needs to be "--group". Also make sure you use your username
in "--allow-principal User:<your username>" instead of "mike". 
    /usr/hdp/current/kafka-broker/bin/ --authorizer
--authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:storm-metron_cluster
--allow-principal User:mike --topic jsonMap
    /usr/hdp/current/kafka-broker/bin/ --authorizer-properties zookeeper.connect=node1:2181
--group jsonMap_parser --list --topic jsonMap
    13. Setup an additional config for the Kafka consumer security protocol options. Create
a file wherever is convenient and note the path. e.g. Here is my config output from root home.
    # cat ~/extra_config.json
    14. Configure the jsonMap parser to handle Kerberos. The parserConfig key is the new piece
required for Kerberos.
    # cat /usr/metron/0.3.1/config/zookeeper/parsers/jsonMap.json
      "parserConfig" : {
          "kafka.producerConfigs" : {
              "security.protocol" : "PLAINTEXTSASL"
          "error.kafka.producerConfigs" : {
              "security.protocol" : "PLAINTEXTSASL"
    15. Push the updated configs to Zookeeper
    /usr/metron/0.3.1/bin/ -z node1:2181 -m PUSH -i /usr/metron/0.3.1/config/zookeeper
    16. Start the topology
    ./ -k node1:6667 -z node1:2181 -s jsonMap -esc ~/extra_config.json
    17. Create a file with some test data that we'll push into the parser Kafka topic
    # cat ~/sampledata.txt
    { "string" : "bar", "number" : 2, "ignored" : [ "blah" ] }
    { "number" : 7 }
    18. Push data into Kafa
    cat ~/sampledata.txt | /usr/hdp/current/kafka-broker/bin/ --broker-list
node1:6667 --security-protocol PLAINTEXTSASL --topic jsonMap
    19. Verify our data comes out on the other end by looking at the enrichments topic.
    /usr/hdp/current/kafka-broker/bin/ --zookeeper node1:2181 --from-beginning
--security-protocol PLAINTEXTSASL --topic enrichments
    The above process worked for me with this PR. That should be it.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message