metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Re: [DISCUSS] Error Indexing
Date Fri, 20 Jan 2017 21:01:13 GMT
As someone who currently fills the platform engineer role, I can give this
idea a huge +1.  My thoughts:

1.  I think it depends on exactly what data is pushed into the index (#3).
However, assuming the errors you proposed recording, I can't see huge
benefits to having more than one dashboard.  I would be happy to be
persuaded otherwise.

2.  I would say yes, storing the errors in HDFS in addition to indexing is
a good thing.  Using METRON-510
<> as a case study, there
is the potential in this environment for attacker-controlled data to result
in processing errors which could be a method of evading security
monitoring.  Once an attack is identified, the long term HDFS storage would
allow better historical analysis for low-and-slow/persistent attacks (I'm
thinking of a method of data exfil that also won't successfully get stored
in Lucene, but is hard to identify over a short period of time).
 - Along this line, I think that there are various parts of Metron (this
included) which could benefit from having method of configuring data aging
by bucket in HDFS (Following Nick's comments here

3.  I would potentially add a hash of the content that failed validation to
help identify repeats over time with less of a concern that you'd have back
to back failures (i.e. instead of storing the value itself).  Additionally,
I think it's helpful to be able to search all times there was an indexing
error (instead of it hitting the catch-all).


On Fri, Jan 20, 2017 at 1:17 PM James Sirota <> wrote:

We already have a capability to capture bolt errors and validation errors
and pipe them into a Kafka topic.  I want to propose that we attach a
writer topology to the error and validation failed kafka topics so that we
can (a) create a new ES index for these errors and (b) create a new Kibana
dashboard to visualize them.  The benefit would be that errors and
validation failures would be easier to see and analyze.

I am seeking feedback on the following:

- How granular would we want this feature to be?  Think we would want one
index/dashboard per source?  Or would it be better to collapse everything
into the same index?
- Do we care about storing these errors in HDFS as well?  Or is indexing
them enough?
- What types of errors should we record?  I am proposing:

For error reporting:
--Message failed to parse
--Enrichment failed to enrich
--Threat intel feed failures
--Generic catch-all for all other errors

For validation reporting:
--What part of message failed validation
--What stellar validator caused the failure

Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org



Sent from my mobile device

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message