metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From justinleet <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Date Fri, 07 Apr 2017 12:45:59 GMT
GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/518

    METRON-799: The MPack should function in a kerberized cluster

    ## Contributor Comments
    Allows the Ambari Kerberos wizard to handle Metron setup.
    
    Changes include:
    - Creation of Keytabs
    - Running everything as the Metron user, including Storm topology workers (on a Kerberized
cluster).
    - Setup for Metron user to actually be able to run (client_jaas setup, home Storm dir
setup, etc.)
    - Adjusting perms to 0755.  The exception is the HDFS output folder on a non-kerb cluster
is left as 0775 because we don't have Storm running workers as metron user on.  When Kerberizing,
the permissions will be restricted down to 0755.
    - Kafka ACLs
    - HBase ACLs
    - Refactored Topic creation to use a common function so I didn't have to edit the same
thing 3 times.
    - Automated updating of Storm configs (the AutoTGT and running workers as user)
    
    There's still more testing I want to do, but this is definitely far enough along to submit
a PR.
    
    I've spun this up on full dev with the now modified Kerberos setup instructions, with
the caveat that Ambari's Storm service check fails (it's harmless, as far as I can tell).
 See below for more details.  As this does not touch the sensors, data will need to be pushed
manually (same as the old instructions).  I've been able to push data from Kafka to Elasticsearch/HDFS.
    
    ### The Bad News
    I would love insight on a problem, if anybody has some.  I haven't edited the docs to
reflect this yet, in the hopes it'll be resolved.
    
    Storm's service check will fail during (and after) Kerberization.  Metron can immediately
be started perfectly fine.  Nothing is legit wrong, but this setup means that the storm user
is unable to submit to the cluster (it doesn't have it's home directory setup with some configs).
 Unfortunately, Ambari runs the service check as the storm user.
    
    This can be worked around by creating ~storm/.storm/storm.yaml
    ```
    nimbus.seeds : ['node1']
    java.security.auth.login.config : '/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf'
    storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
    ```
    `java.security.auth.login.conf` can also be `/etc/storm/conf/storm_jaas.conf`, but the
value above leads me to my next point.  All of these values already exist in storm.yaml. 
The fact that they need to be specified again in the user's home is really strange. And it'll
give an error that the TGT found is not renewable, not something you'd expected.
    
    I'm unsure if there are restrictions on where Ambari chooses to run service check, so
it's possible this would have to be setup on every node Storm lives on the cluster. I'm also
unsure if we can actually have Ambari automate this if it turns out to be necessary, since
we aren't the Storm service.
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron (Incubating).  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines
and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).

    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically
master)?
    
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested
manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the
root incubating-metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - ~Have you written or updated unit tests and or integration tests to verify your changes?~
    - ~If adding new dependencies to the code, are these dependencies licensed in a way that
is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~

    - [x] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered
by building and verifying the site-book? If not then run the following commands and the verify
changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      bin/generate-md.sh
      mvn site:site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible.
    It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal
repository such that your branches are built there before submitting a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron mpack-kerberos

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #518
    
----
commit 7934bec0c8f302bba3f68b6afaaf89271b0efce5
Author: justinleet <justinjleet@gmail.com>
Date:   2017-03-15T18:55:06Z

    hbase kinit for ambari
    
    * hbase kinit
    
    * Adding kinits around kafka topic creation
    
    * adding in hbase acl
    
    * All status checks, etc. seem to work now
    
    * Default to PLAINTEXT

commit fe065f2ea39c3146a60fe23504655089309f5481
Author: justinleet <justinjleet@gmail.com>
Date:   2017-03-15T19:42:51Z

    Splitting out ACLs from Kafka and HBase setup (#3)

commit ee8d35dc5dcdaee507ff906aa83e83033d0606ea
Author: justinleet <justinjleet@gmail.com>
Date:   2017-03-15T20:41:56Z

    Kerb testing (#4)
    
    * Splitting out ACLs from Kafka and HBase setup
    
    * Moving user outside

commit 7c0c5588e1eb4100fe455c199b0729088f47546f
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-06T02:40:59Z

    mostly working

commit f75b4c37e6f4c7e72ca5cdd22a4f8b830db3ecc3
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-06T12:49:14Z

    working now again after some refactoring.

commit 91a89429201d3d45ece33dc3c36ee9c6a34ebbb6
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-06T18:59:21Z

    fixes, refactoring, and improvements

commit fd5238f632a9d4fc62a3e59ca8d3bb24e2e3c55b
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-06T19:23:31Z

    Cleanup

commit 569be68b17d0e84ece51015140e6eade755f3cb5
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-07T12:09:28Z

    error topics plus fixes plus documentation

commit 5418a2eb914faac0450f7749b6c76f75a2da4e19
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-07T12:22:35Z

    Removing leftover TODO

commit 049cf524f919f608dbce2b6c5dacbd4552ce3efb
Author: justinjleet <justinjleet@gmail.com>
Date:   2017-04-07T12:37:12Z

    Caught a testing change in the last git add

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message