metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Allen (JIRA)" <>
Subject [jira] [Commented] (METRON-1832) Recurrent Large Indexing Error Messages
Date Fri, 19 Oct 2018 17:13:00 GMT


Nick Allen commented on METRON-1832:

As you can see from the attached error message, the escape character '\' also continually
compound one another leading to long strings of '\' characters.
  \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"All datanodes DatanodeInfoWithStorage[,DS-495f88fc-f 
  d24-41e1-b55c-fc6ec586c6e4,DISK] are bad. Aborting...\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\ 

> Recurrent Large Indexing Error Messages
> ---------------------------------------
>                 Key: METRON-1832
>                 URL:
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Nick Allen
>            Priority: Major
>         Attachments: error-message.json
> If any index destination like HDFS, Elasticsearch, or Solr goes down while the Indexing
topology is running, an error message is created and sent back to the user-defined error topic. 
By default, this is defined to also be the 'indexing' topic.
> The Indexing topology then consumes this error message and attempts to write it again.
If the index destination is still down, another error occurs and another error message is
created that encapsulates the original error message.  That message is then sent to the 'indexing'
topic, which is later consumed, yet again, by the Indexing topology.
> These error messages will continue to be recycled and grow larger and larger as each
new error message encapsulates all previous error messages in the "raw_message" field.
> Once the index destination recovers, one giant error message will finally be written
that contains massively duplicated, useless information which can further negatively impact
performance of the index destination.
> See attachment for example of once of these error messages.

This message was sent by Atlassian JIRA

View raw message