metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Allen (Jira)" <j...@apache.org>
Subject [jira] [Commented] (METRON-2284) Metron Profiler for Spark doesn't work as expected
Date Thu, 07 Nov 2019 17:18:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969431#comment-16969431
] 

Nick Allen commented on METRON-2284:
------------------------------------

This definitely looks to be a bug.  The behavior between the Profiler in the REPL and in
the Batch Profiler in Spark should be identical, but it seems not to be. 

What are you trying to do with this profile?  Maybe I can help you with a workaround until
we can fix the problem.

BTW, thank you for providing such a clear bug report with the exact steps to replicate. 
Very helpful!

> Metron Profiler for Spark doesn't work as expected
> --------------------------------------------------
>
>                 Key: METRON-2284
>                 URL: https://issues.apache.org/jira/browse/METRON-2284
>             Project: Metron
>          Issue Type: Bug
>    Affects Versions: 0.7.1
>            Reporter: Maxim Dashenko
>            Priority: Major
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.metron.profiler.spark.cli.BatchProfilerCLI
--properties-file /usr/hcp/current/metron/config/batch-profiler.properties ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar
--config /usr/hcp/current/metron/config/batch-profiler.properties --profiles ~/profiler.json
> {code}
>  cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>  
> cat ~/profiler.json
> {code}
> {
>  "profiles":[
>    {
>      "profile":"batchtest5",
>      "onlyif":"source.type == 'testsource' and devicehostname == 'windows9.something.com'",
>      "foreach":"devicehostname",
>      "init":{
>        "val":"SET_INIT()"
>      },
>      "update":{
>        "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
>      },
>     "result":{
>        "profile":"val"
>     }
>    }
>  ],
>  "timestampField":"timestamp"
> }
> {code}
>  cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000, "source.type":
"testsource"}
> {code}
> Stellar statement
> {code}
> PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100, 'DAYS'))
> {code}
> Returns:
> {code}
> [[true]]
> {code}
> Expected result:
> {code}
> [[false]]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message