metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] metron pull request #1243: METRON-1831 Project Version Substitution Not Work...
Date Fri, 19 Oct 2018 22:21:13 GMT
GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/1243

    METRON-1831 Project Version Substitution Not Working

    When running the stellar REPL script, a warning is reported.
    ```
    bin/stellar: line 36: ${project.version}: bad substitution
    ```
    For example...
    ```
    [root@node1 ~]# source /etc/default/metron
    [root@node1 ~]# cd $METRON_HOME
    [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
    bin/stellar: line 36: ${project.version}: bad substitution
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    Stellar, Go!
    Functions are loading lazily in the background and will be unavailable until loaded fully.
    {es.clustername=metron, es.ip=node1:9300, es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing,
update.hbase.table=metron_update, update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s},
es.document.id=, profiler.client.period.duration=15, profiler.client.period.duration.units=MINUTES,
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, bootstrap.servers=node1:6667,
source.type.field=source:type, threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15,
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, profiler.writer.batchTimeout=0,
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}
    ```
    ## Problem
    
    In the stellar REPL launch script, the ${project.version} is not getting substituted at
build time, perhaps because it is wrapped inside another ${...}.  This is probably why it
is an issue in the stellar script and not the other scripts.
    
    ## Change 
    
    * The stellar REPL launch script was changed in #1232 , but I did not see this warning
occur when testing that change.  
    * Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` automatically.
There really is no need to set `METRON_HOME` and `METRON_VERSION` in the script any longer.
 It is cleaner to just remove this (which I almost did in #1232).  
    * I use a `set -u` so the script will error if `METRON_HOME` is not defined by some odd
happenstance.
    
    ## Testing
    
    1. Stand-up a development environment.
    
    1. Launch the Stellar REPL.  Ensure there is no warning reported.
        ```
        [vagrant@node1 ~]$ sudo su -
        [root@node1 ~]# cd /usr/metron/0.6.1/
        [root@node1 0.6.1]# bin/stellar
        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
        Stellar, Go!
        Functions are loading lazily in the background and will be unavailable until loaded
fully.
        {}
        [Stellar]>>>
        ```
    1. Ensure the standard suite of Stellar functions are discovered.
    
        ```
        [Stellar]>>> %functions
        ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, BYTEARRAY_MATCHER,
CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH,
DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD,
DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, ENRICHMENT_STELLAR_TRANSFORM_ADD,
ENRICHMENT_STELLAR_TRANSFORM_PRINT, ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT,
FILTER, FLOOR, FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, GEOHASH_FROM_LATLONG,
GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE,
GET_LAST, GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, HDFS_READ_LINES,
HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, INDEXING_SET_BATCH,
INDEXING_SET_ENABLED, INDEXING_SET_INDEX, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY,
IS_E
 NCODING, IS_INTEGER, IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JOIN, KAFKA_FIND, KAFKA_GET, KAFKA_PROPS,
KAFKA_PUT, KAFKA_SEEK, KAFKA_TAIL, LENGTH, LIST_ADD, LN, LOCAL_LS, LOCAL_READ, LOCAL_READ_LINES,
LOCAL_RM, LOCAL_WRITE, LOG10, LOG2, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS,
MAP_GET, MAX, MIN, MONTH, MULTISET_ADD, MULTISET_INIT, MULTISET_MERGE, MULTISET_REMOVE, MULTISET_TO_SET,
OBJECT_GET, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PARSER_STELLAR_TRANSFORM_ADD,
PARSER_STELLAR_TRANSFORM_PRINT, PARSER_STELLAR_TRANSFORM_REMOVE, PREPEND_IF_MISSING, PROFILER_APPLY,
PROFILER_FLUSH, PROFILER_INIT, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME,
REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, REGEXP_REPLACE, ROUND, SAMPLE_ADD, SAMPLE_GET, SAMPLE_INIT,
SAMPLE_MERGE, SET_ADD, SET_INIT, SET_MERGE, SET_REMOVE, SHELL_EDIT, SHELL_GET_EXPRESSION,
SHELL_LIST_VARS, SHELL_MAP2TABLE, SHELL_VARS2MAP, SIN, SPLIT, SQRT, STARTS_WITH, STATS_ADD,
STATS_BIN, STATS_COUNT, STAT
 S_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN,
STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS,
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SUBSTRING, SYSTEM_ENV_GET,
SYSTEM_PROPERTY_GET, TAN, THREAT_TRIAGE_ADD, THREAT_TRIAGE_CONFIG, THREAT_TRIAGE_INIT, THREAT_TRIAGE_PRINT,
THREAT_TRIAGE_REMOVE, THREAT_TRIAGE_SCORE, THREAT_TRIAGE_SET_AGGREGATOR, TLSH_DIST, TO_DOUBLE,
TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_JSON_LIST, TO_JSON_MAP, TO_JSON_OBJECT, TO_LONG,
TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL,
WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST
        ```
    
    ## Pull Request Checklist
    
    - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically
master)?
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested
manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the
root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way
that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1831

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1243.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1243
    
----
commit 020fca5402a40f2acccb5f80d4295368bf3761de
Author: Nick Allen <nick@...>
Date:   2018-10-19T22:00:59Z

    METRON-1831 Project Version Substitution Not Working

----


---

Mime
View raw message