metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <>
Subject [DISCUSS] Error Indexing
Date Fri, 20 Jan 2017 18:16:57 GMT
We already have a capability to capture bolt errors and validation errors and pipe them into
a Kafka topic.  I want to propose that we attach a writer topology to the error and validation
failed kafka topics so that we can (a) create a new ES index for these errors and (b) create
a new Kibana dashboard to visualize them.  The benefit would be that errors and validation
failures would be easier to see and analyze.  

I am seeking feedback on the following:

- How granular would we want this feature to be?  Think we would want one index/dashboard
per source?  Or would it be better to collapse everything into the same index?
- Do we care about storing these errors in HDFS as well?  Or is indexing them enough?
- What types of errors should we record?  I am proposing:

For error reporting:
--Message failed to parse
--Enrichment failed to enrich
--Threat intel feed failures 
--Generic catch-all for all other errors

For validation reporting:
--What part of message failed validation
--What stellar validator caused the failure

Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

View raw message