metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Girish N <giri.narasimha.mur...@gmail.com>
Subject Re: Exceptions in Parser, enrichments and indexing topology
Date Wed, 06 Sep 2017 11:02:39 GMT
Hello,

Grok pattern was wrong, changed it as below and tested and got a match.
Sample Log - Sep  6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.

Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
Grok pattern 2 - %{SYSLOGLINE}

Tried parsing with both the grok patterns, but still facing the same
exceptions.

Regards
Girish N


On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian <
asubramanian@hortonworks.com> wrote:

> Hey Girish,
>
> Can you validate using http://grokconstructor.appspot.com/do/match that
> you can parse the sample message using the pattern? I tried with your
> example and it did not work.
>
> -Anand
>
>
>
> On 9/6/17, 4:06 PM, "Girish N" <giri.narasimha.murthy@gmail.com> wrote:
>
> >Thanks for your response Otto Fowler,
> >
> >I tried with the below config. Still the same exception. Kindly let me
> know
> >if anything else has to be changed.
> >
> >{
> >"parserClassName": "org.apache.metron.parsers.GrokParser",
> >"sensorTopic": "log",
> >"parserConfig": {
> >"grokPath": "/patterns/log",
> >"patternLabel": "SYS_DELIMITED"
> >}}
> >
> >java.lang.IllegalStateException: Unhandled bulk errors in response:
> >{java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >parserBolt:4, stream: error, id: {},
> >[{"exception":"java.lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {} . Check the pattern at: \/patterns\/log on
> >Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> >Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep  6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {}
> >
> >
> >Regards
> >Girish N
> >
> >
> >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
> >
> >> Change the grokPath to match below and try.
> >>
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >>
> >> This path is not an absolute path.
> >>
> >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> >> com) wrote:
> >>
> >> Hi,
> >>
> >> I am trying to parse the syslog I am getting below exceptions. Kindly
> help
> >> to resolve the issue. Thanks
> >>
> >>
> >> Sample Syslog-
> >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
> >>
> >>
> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> >> SYS_DELIMITED
> >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY}
> )?
> >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
> >>
> >> 2. Created a Parser config log.json
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "usr/metron/0.4.0/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >> 3. Created a indexing config log.json
> >> {
> >> "hdfs" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "elasticsearch" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "solr" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> }
> >> }
> >>
> >> 4. Pushed and Dumped using
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> >> /usr/metron/0.4.0/config/zookeeper
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
> >>
> >> 5. Then started the parser topology , getting the below exceptions in
> log
> >> worker-artifacts (
> >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
> >>
> >> java.lang.IllegalStateException: Unhandled bulk errors in response:
> >> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >> parserBolt:4, stream: error, id: {},
> >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> >> statement produced a null message. Original message was: Sep 6 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6
> 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS
> >> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >>
> >> Grok parser Error: Grok statement produced a null message. Original
> >> message
> >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >> and the parsed message was: {}
> >>
> >> 6. For the enrichment topology, getting the below exception.
> >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The
> configuration
> >> request.required.acks = 1 was supplied but isn't a known config.
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> >> 0.10.0.1
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> >> a7a17cdec9eaa6c5
> >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentErrorOutputBolt:(1)
> >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> >> System classloader
> >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> >> threatIntelSplitBolt:(14)
> >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentSplitBolt:(3)
> >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> >> hostEnrichmentBolt:(5)
> >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentJoinBolt:(2)
> >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> >> 'enrichments': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> >> at
> >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.base.ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:521)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> initForReading(ObjectMapper.java:3847)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> readMapAndClose(ObjectMapper.java:3792)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper.
> >> readValue(ObjectMapper.java:2867)
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> >> ~[stormjar.jar:?]
> >>
> >> 7. For the Indexing topology , getting below exception
> >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.
> >> JsonParser._constructError(JsonParser.java:1581)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.base.
> ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:533)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:
> 3451)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._nextTokenNotInObject(
> UTF8StreamJsonParser.java:841)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser.
> >> nextToken(UTF8StreamJsonParser.java:737)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2874)
> >>
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.common.configuration.IndexingConfigurations.
> >> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> >> ~[stormjar.jar:?]
> >>
> >>
> >> Kindly help to resolve the issue.
> >>
> >> Regards
> >> Girish N
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message