metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Subramanian <asubraman...@hortonworks.com>
Subject Re: Exceptions in Parser, enrichments and indexing topology
Date Wed, 06 Sep 2017 10:47:41 GMT
Hey Girish,

Can you validate using http://grokconstructor.appspot.com/do/match that you can parse the
sample message using the pattern? I tried with your example and it did not work.

-Anand



On 9/6/17, 4:06 PM, "Girish N" <giri.narasimha.murthy@gmail.com> wrote:

>Thanks for your response Otto Fowler,
>
>I tried with the below config. Still the same exception. Kindly let me know
>if anything else has to be changed.
>
>{
>"parserClassName": "org.apache.metron.parsers.GrokParser",
>"sensorTopic": "log",
>"parserConfig": {
>"grokPath": "/patterns/log",
>"patternLabel": "SYS_DELIMITED"
>}}
>
>java.lang.IllegalStateException: Unhandled bulk errors in response:
>{java.lang.IllegalArgumentException: Topic cannot be null=[source:
>parserBolt:4, stream: error, id: {},
>[{"exception":"java.lang.IllegalStateException:
>Grok parser Error: Grok statement produced a null message. Original message
>was: Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>and the parsed message was: {} . Check the pattern at: \/patterns\/log on
>Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
>Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
>Grok parser Error: Grok statement produced a null message. Original message
>was: Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>and the parsed message was: {}
>
>
>Regards
>Girish N
>
>
>On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ottobackwards@gmail.com> wrote:
>
>> Change the grokPath to match below and try.
>>
>> {
>> "parserClassName": "org.apache.metron.parsers.GrokParser",
>> "sensorTopic": "log",
>> "parserConfig": {
>> "grokPath": "/patterns/log",
>> "patternLabel": "SYS_DELIMITED"
>> }}
>>
>>
>> This path is not an absolute path.
>>
>> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
>> com) wrote:
>>
>> Hi,
>>
>> I am trying to parse the syslog I am getting below exceptions. Kindly help
>> to resolve the issue. Thanks
>>
>>
>> Sample Syslog-
>> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
>>
>>
>> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
>> SYS_DELIMITED
>> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
>> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
>>
>> 2. Created a Parser config log.json
>> {
>> "parserClassName": "org.apache.metron.parsers.GrokParser",
>> "sensorTopic": "log",
>> "parserConfig": {
>> "grokPath": "usr/metron/0.4.0/patterns/log",
>> "patternLabel": "SYS_DELIMITED"
>> }}
>>
>> 3. Created a indexing config log.json
>> {
>> "hdfs" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> },
>> "elasticsearch" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> },
>> "solr" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> }
>> }
>>
>> 4. Pushed and Dumped using
>> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
>> /usr/metron/0.4.0/config/zookeeper
>> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
>>
>> 5. Then started the parser topology , getting the below exceptions in log
>> worker-artifacts (
>> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
>>
>> java.lang.IllegalStateException: Unhandled bulk errors in response:
>> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
>> parserBolt:4, stream: error, id: {},
>> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
>> statement produced a null message. Original message was: Sep 6 12:41:02
>> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
>> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
>> exza-ThinkPad-X240 systemd[1]: Started CUPS
>> Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
>>
>> Grok parser Error: Grok statement produced a null message. Original
>> message
>> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>> and the parsed message was: {}
>>
>> 6. For the enrichment topology, getting the below exception.
>> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
>> request.required.acks = 1 was supplied but isn't a known config.
>> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
>> 0.10.0.1
>> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
>> a7a17cdec9eaa6c5
>> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentErrorOutputBolt:(1)
>> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
>> System classloader
>> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
>> threatIntelSplitBolt:(14)
>> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentSplitBolt:(3)
>> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
>> hostEnrichmentBolt:(5)
>> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentJoinBolt:(2)
>> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
>> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
>> 'enrichments': was expecting ('true', 'false' or 'null')
>> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
>> at
>> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)
>>
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.base.ParserMinimalBase._
>> reportError(ParserMinimalBase.java:521)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> reportInvalidToken(UTF8StreamJsonParser.java:3466)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
>>
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper._
>> initForReading(ObjectMapper.java:3847)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper._
>> readMapAndClose(ObjectMapper.java:3792)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper.
>> readValue(ObjectMapper.java:2867)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
>> ~[stormjar.jar:?]
>>
>> 7. For the Indexing topology , getting below exception
>> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
>> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
>> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.
>> JsonParser._constructError(JsonParser.java:1581)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._
>> reportError(ParserMinimalBase.java:533)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.
>> nextToken(UTF8StreamJsonParser.java:737)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
>>
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.common.configuration.IndexingConfigurations.
>> updateSensorIndexingConfig(IndexingConfigurations.java:52)
>> ~[stormjar.jar:?]
>>
>>
>> Kindly help to resolve the issue.
>>
>> Regards
>> Girish N
>>
>>
Mime
View raw message