metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #510: METRON-821 Minor fixes in full dev kerbe...
Date Thu, 13 Apr 2017 20:42:52 GMT
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/510#discussion_r111482378
  
    --- Diff: metron-deployment/vagrant/Kerberos-setup.md ---
    @@ -87,147 +86,159 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
         ![enable keberos configure](readme-images/custom-storm-site-final.png)
     
     9. Setup Metron keytab
    -  ```
    -kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    -cp metron.headless.keytab /etc/security/keytabs && \
    -chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    -chmod 440 /etc/security/keytabs/metron.headless.keytab
    -  ```
    +    ```
    +    kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
    +    cp metron.headless.keytab /etc/security/keytabs && \
    +    chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
    +    chmod 440 /etc/security/keytabs/metron.headless.keytab
    +    ```
     
     10. Kinit with the metron user
    -  ```
    -kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    -  ```
    +    ```
    +    kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
    +    ```
     
     11. First create any additional Kafka topics you will need. We need to create the topics
before adding the required ACLs. The current full dev installation will deploy bro, snort,
enrichments, and indexing only. e.g.
    -  ```
    -${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic
yaf --partitions 1 --replication-factor 1
    -  ```
    +    ```
    +    ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create
--topic yaf --partitions 1 --replication-factor 1
    +    ```
     
     12. Setup Kafka ACLs for the topics
    -  ```
    -export KERB_USER=metron;
    -for topic in bro enrichments indexing snort; do
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--topic ${topic};
    -done;
    -  ```
    +    ```
    +    export KERB_USER=metron;
    +    for topic in bro enrichments indexing snort; do
    +        ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--topic ${topic}
    +    done
    +    ```
     
     13. Setup Kafka ACLs for the consumer groups
    -  ```
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group bro_parser;
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group snort_parser;
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group yaf_parser;
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group enrichments;
    -${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group indexing;
    -  ```
    +    ```
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group bro_parser
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group snort_parser
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group yaf_parser
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group enrichments
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--group indexing
    +    ```
     
     14. Add metron user to the Kafka cluster ACL
    -  ```
    -/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--cluster kafka-cluster
    -  ```
    +    ```
    +    ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer
--authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER}
--cluster kafka-cluster
    +    ```
     
     15. We also need to grant permissions to the HBase tables. Kinit as the hbase user and
add ACLs for metron.
    -  ```
    -kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    -echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    -echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
    -  ```
    +    ```
    +    kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
    +    echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
    +    echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
    +    ```
     
     16. Create a “.storm” directory in the metron user’s home directory and switch
to that directory.
    -  ```
    -su metron && cd ~/
    -mkdir .storm
    -cd .storm
    -  ```
    +    ```
    +    su metron
    +    mkdir ~/.storm
    +    cd ~/.storm
    +    ```
     
     17. Create a custom client jaas file. This should look identical to the Storm client
jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client
stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
    -  ```
    -[metron@node1 .storm]$ cat client_jaas.conf
    -StormClient {
    -   com.sun.security.auth.module.Krb5LoginModule required
    -   useTicketCache=true
    -   renewTicket=true
    -   serviceName="nimbus";
    -};
    -Client {
    -   com.sun.security.auth.module.Krb5LoginModule required
    -   useKeyTab=true
    -   keyTab="/etc/security/keytabs/metron.headless.keytab"
    -   storeKey=true
    -   useTicketCache=false
    -   serviceName="zookeeper"
    -   principal="metron@EXAMPLE.COM";
    -};
    -KafkaClient {
    -   com.sun.security.auth.module.Krb5LoginModule required
    -   useKeyTab=true
    -   keyTab="/etc/security/keytabs/metron.headless.keytab"
    -   storeKey=true
    -   useTicketCache=false
    -   serviceName="kafka"
    -   principal="metron@EXAMPLE.COM";
    -};
    -  ```
    +    ```
    +    cat << EOF > client_jaas.conf
    +    StormClient {
    +        com.sun.security.auth.module.Krb5LoginModule required
    +        useTicketCache=true
    +        renewTicket=true
    +        serviceName="nimbus";
    +    };
    +    Client {
    +        com.sun.security.auth.module.Krb5LoginModule required
    +        useKeyTab=true
    +        keyTab="/etc/security/keytabs/metron.headless.keytab"
    +        storeKey=true
    +        useTicketCache=false
    +        serviceName="zookeeper"
    +        principal="metron@EXAMPLE.COM";
    +    };
    +    KafkaClient {
    +        com.sun.security.auth.module.Krb5LoginModule required
    +        useKeyTab=true
    +        keyTab="/etc/security/keytabs/metron.headless.keytab"
    +        storeKey=true
    +        useTicketCache=false
    +        serviceName="kafka"
    +        principal="metron@EXAMPLE.COM";
    +    };
    +    EOF
    +    ```
     
     18. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
    -  ```
    -[metron@node1 .storm]$ cat storm.yaml
    -nimbus.seeds : ['node1']
    -java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    -storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    -  ```
    +    ```
    +    cat << EOF > storm.yaml
    +    nimbus.seeds : ['node1']
    +    java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
    +    storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
    +    EOF
    +    ```
     
     19. Create an auxiliary storm configuration json file in the metron user’s home directory.
Note the login config option in the file points to our custom client_jaas.conf.
    -  ```
    -cd /home/metron
    -[metron@node1 ~]$ cat storm-config.json
    -{
    -  "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    -}
    -  ```
    +    ```
    +    cat << EOF > ~/storm-config.json
    +    {
    +        "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
    +    }
    +    EOF
    +    ```
     
     20. Setup enrichment and indexing.
     
    -    a. Modify enrichment.properties - `${METRON_HOME}/config/enrichment.properties`
    -
    +    a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
         ```
    -    kafka.security.protocol=PLAINTEXTSASL
    -    topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +    if [[ $EUID -ne 0 ]]; then
    +        echo -e "\nERROR:\tYou must be root to run these commands.  You may need to type
exit."
    +    else
    +        sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/'
${METRON_HOME}/config/enrichment.properties
    +        sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
${METRON_HOME}/config/enrichment.properties
    +    fi
         ```
     
    -    b. Modify elasticsearch.properties - `${METRON_HOME}/config/elasticsearch.properties`
    -
    +    b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
         ```
    -    kafka.security.protocol=PLAINTEXTSASL
    -    topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
    +    if [[ $EUID -ne 0 ]]; then
    +        echo -e "\nERROR:\tYou must be root to run these commands.  You may need to type
exit."
    +    else
    +        sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/'
${METRON_HOME}/config/elasticsearch.properties
    +        sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/'
${METRON_HOME}/config/elasticsearch.properties
    +    fi
    --- End diff --
    
    I kind of prefer just showing the key=value so that it is clear to the user what properties
need to change.
    
    I appreciate the fact that you've automated it, but since these are manual instructions,
maybe we should opt for making them as clear as possible. I could be convinced otherwise though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message