metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <>
Subject [GitHub] metron pull request #1278: METRON-1892 Parser Debugger Should Load Config Fr...
Date Mon, 10 Dec 2018 00:43:59 GMT
GitHub user nickwallen reopened a pull request:

    METRON-1892 Parser Debugger Should Load Config From Zookeeper

    When using the parser debugger functions created in #1265, the user has to manually specify
the parser configuration.  This is useful when testing out a new parser before it goes live.
    In other cases, it would be simpler to use the sensor configuration values that are 'live'
and already loaded in Zookeeper.  For example, I might want to test why a particular messages
fails to parse in my environment.
    ## Try It Out
    Try out the following examples in the Stellar REPL.
    1. Launch a development environment.
    1. Launch the REPL.
        source /etc/default/metron
        cd $METRON_HOME
        bin/stellar -z $ZOOKEEPER
    ### Parse a Message
    1. Grab a message from the input topic to parse. You could also just mock-up a message
that you would like to test.
        [Stellar]>>> input := KAFKA_GET('bro')
        [{"http": {"ts":1542313125.807068,"uid":"CUrRne3iLIxXavQtci","id.orig_h"...
    1. Initialize the parser. The parser configuration for 'bro' will be loaded automatically
from Zookeeper.
        [Stellar]>>> parser := PARSER_INIT("bro")
        Parser{0 successful, 0 error(s)}
    1. Parse the message.
        [Stellar]>>> msgs := PARSER_PARSE(parser, input)
    1. The parser will tally the success.
        [Stellar]>>> parser
        Parser{1 successful, 0 error(s)}
    1. Review the successfully parsed message.
        [Stellar]>>> LENGTH(msgs)
        [Stellar]>>> msg := GET(msgs, 0)
        [Stellar]>>> MAP_GET("guid", msg)
        [Stellar]>>> MAP_GET("timestamp", msg)
        [Stellar]>>> MAP_GET("source.type", msg)
    ### Missing Configuration
    1. If the configuration does not exist in Zookeeper, you should see something like this.

        I have not configured a parser named 'tuna' in my environment (but I could go for
a tuna sandwich right about now ).
        [Stellar]>>> bad := PARSER_INIT('tuna')
        [!] Unable to parse: PARSER_INIT('tuna') due to: Unable to read configuration from
Zookeeper; sensorType = tuna
        org.apache.metron.stellar.dsl.ParseException: Unable to parse: PARSER_INIT('tuna')
due to: Unable to read configuration from Zookeeper; sensorType = tuna
        	at org.apache.metron.stellar.common.BaseStellarProcessor.createException(
        	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(
        	at java.util.concurrent.ThreadPoolExecutor$
        Caused by: java.lang.IllegalArgumentException: Unable to read configuration from Zookeeper;
sensorType = tuna
        	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(
        	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(
        	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(
        	... 9 more
    ## Pull Request Checklist
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically
    - [x] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested
    - [x] Have you ensured that the full suite of tests and checks have been executed in the
root metron folder via:
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way
that is compatible for inclusion under [ASF 2.0](
    - [x] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?

You can merge this pull request into a Git repository by running:

    $ git pull METRON-1892

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1278
commit b56e736bafe93dd6640be90ab3b144ff7f7a918a
Author: Nick Allen <nick@...>
Date:   2018-11-21T16:43:39Z

    METRON-1892 Parser Debugger Should Load Config From Zookeeper

commit cc29b6d378e5b590b146ba6aba4268dd449789a8
Author: Nick Allen <nick@...>
Date:   2018-12-08T16:21:39Z

    Merge remote-tracking branch 'apache/master' into METRON-1892



View raw message