metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: Issues with Metron profiler
Date Tue, 12 Mar 2019 21:35:33 GMT
Hi Anil,

Can you share your profile definition? For the STATS_BIN issue you're
seeing, it seems like your bins are wrong, however you're using the
pre-defined DECILE bin list, which is unit tested (and the tests are
passing). It may be that the stats object isn't quite right, but I can't
tell from this snippet alone. Your activity_count at least appears
populated.

On Mon, Mar 11, 2019 at 6:16 PM Anil Donthireddy <Anil.Donthireddy@sstech.us>
wrote:

> Hi,
>
>
>
> I am seeing below issues with Metron profiler.
>
>
>
> 1)      Profiler is writing the duplicate data to Hbase. For example I
> have inserted few records couple of days back to my test sensor topic and
> written a profile that counts the number of records. I see that profiler is
> still storing the same count continuously in to Hbase. As I observe when I
> see the WARN message below in profiler log, I see the profiler inserts data
> to Hbase that number of times (I am not sure if there is any relation)
>
> a.       2019-03-07 15:56:28.419 o.a.m.p.DefaultMessageDistributor
> pool-17-thread-1 [WARN] Profile expired from active cache;
> profile=demo_user_activity_count_ar, entity=Test1User5
>
> 2)      I am storing the STAT objects to Hbase and trying to calculate
> the bin in another profiler by pulling Stats from Hbase. I see below
> exception after a while
>
> a.       2019-03-11 11:24:51.039 o.a.m.p.DefaultProfileBuilder
> pool-18-thread-1 [ERROR] Bad 'update' expression: error='Unable to parse:
> if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to:
> Your bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'
>
> b.      org.apache.metron.stellar.dsl.ParseException: Unable to parse: if
> EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your
> bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24
>
>
>
> I would like to understand what does it meant by “[WARN] Profile expired
> from active cache; profile=demo_user_activity_count_ar, entity=Test1User5”,
> when can this happen  and how to resolve this?
>
>
>
> I would also like to understand how to resolve the issue “[ERROR] Bad
> 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s,
> activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing
> with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'”
>
>
>
> Looking forward for some help from Metron team or community to understand
> and resolve the above issues.
>
>
>
> Thanking you.
>
>
>
> Thanks,
>
> Anil.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message