metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (METRON-694) Index Errors from Topologies
Date Wed, 01 Mar 2017 14:10:45 GMT


ASF GitHub Bot commented on METRON-694:

Github user merrimanr commented on the issue:
    The latest commit includes the changes discussed.  Most error messages now go to the indexing
topic by default so there is no need for another error indexing topology.  The error topic
setting for the enrichment and indexing topologies are already exposed through flux properties
so I left them there.  I added a "parser.error.topic" property to the global config for parser
topologies (should indexing topic be the default if that property is missing or should we
let it fail?).
    For the concern about indexing errors getting in a loop, I thought of a simple solution
that required no extra work.  If an error happens in the indexing topology, simply send it
to a different error topic.  Would this be acceptable?  I wanted to throw this out there before
implementing custom logic in the indexing topology classes.
    The error message structure did change slightly.  In my testing I realized that the type
for the "raw_message" field could be different (byte[] vs JSON) depending on where the error
happened (pre vs post parse).  If an error message with byte[] as raw_message was indexed
first, Elasticsearch would error on subsequent error messages with JSON as raw_message.  To
address this, I serialized JSON raw messages before storing them in this field.  Now raw_message
is always and string and raw_message_bytes is always a byte array.
    All tests are passing and it has been tested in quick dev.  Some tips on testing the different
    - the mock Bro sensor in quick dev contains a couple unparsable messages so parser errors
should show up in the error index by default
    - Invalid parser errors can be generated by adding a field validation to the global config
that will always fail 
    - for testing indexing errors, copy a message from the indexing topic, change a field
with number type (ip_src_port for example) to a non-number string, then put that message back
into the indexing topic
    @cestella, would be good to get your opinion on the MessageGetter classes and where that
ended up.  Not sure I love the syntax for default message getters.
    The MPack changes are now minimal but it should still be tested there and that test is
pending.  The documentation still needs to be updated so please don't merge this until that
is done.  

> Index Errors from Topologies
> ----------------------------
>                 Key: METRON-694
>                 URL:
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Ryan Merriman
> Need to make sure (and review) that all the bolts write into the error queue. Errors
should then be consumed from the error queue and indexed.

This message was sent by Atlassian JIRA

View raw message