metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mmiklavc <...@git.apache.org>
Subject [GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...
Date Thu, 04 Jan 2018 23:10:58 GMT
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/831
  
    I'm looking at this deeper and thinking through how status_params.py works with the config
files. I'm looking for a dynamic way to manage the endpoints, in the future (not for this
PR). For instance, we currently have this to load the config (in master)
    
    ```
    # common-services/METRON/CURRENT/package/scripts/params/params_linux.py
    # Indexing
    indexing_kafka_start = config['configurations']['metron-indexing-env']['indexing_kafka_start']
    ```
    This PR currently sets `ra_indexing_kafka_start` as the property name and `Elasticsearch
Indexing Offset` as the display name. I think this is fine, but I'm unclear how best to manage
1..n indexing endpoints in later PR's because I don't know that Ambari (yet) offers a way
to have dynamic config based on the type of service you choose to install. Maybe we can leverage
a dropdown option as part of metron-env that we then leverage to dynamically choose the indexing
config type you selected. So metron-env has the following:
    
    ```
    <property>
            <name>ra_indexing_framework</name>
            <description>How you like to index for RA, friend?</description>
            <value>Elasticsearch</value>
            <display-name>Indexing Framework for Random Access</display-name>
            <value-attributes>
                <type>value-list</type>
                <entries>
                    <entry>
                        <value>Elasticsearch</value>
                    </entry>
                    <entry>
                        <value>Solr</value>
                    </entry>
                </entries>
                <selection-cardinality>1</selection-cardinality>
            </value-attributes>
        </property>
    ```
    
    and the earlier snippet above looks something like:
    
    ```
    # common-services/METRON/CURRENT/package/scripts/params/params_linux.py
    # Indexing
    indexing_kafka_start = config['configurations'][ra_indexing_framework + '-env']['indexing_kafka_start']
    ```
    
    Not sure if you can do that, just thinking out loud. Also, this handles the ability to
plug in multiple providers from which you can select only one, but it does not handle the
1..n providers scenario simultaneously, if there were such a need.


---

Mime
View raw message