metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: Exceptions in Parser, enrichments and indexing topology
Date Wed, 06 Sep 2017 10:15:02 GMT
Change the grokPath to match below and try.

{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}


This path is not an absolute path.

On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.com)
wrote:

Hi,

I am trying to parse the syslog I am getting below exceptions. Kindly help
to resolve the issue. Thanks


Sample Syslog-
Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.


1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
SYS_DELIMITED
<%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}

2. Created a Parser config log.json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "usr/metron/0.4.0/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}

3. Created a indexing config log.json
{
"hdfs" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "log",
"batchSize": 5,
"enabled" : true
}
}

4. Pushed and Dumped using
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
/usr/metron/0.4.0/config/zookeeper
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP

5. Then started the parser topology , getting the below exceptions in log
worker-artifacts (
usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)

java.lang.IllegalStateException: Unhandled bulk errors in response:
{java.lang.IllegalArgumentException: Topic cannot be null=[source:
parserBolt:4, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
statement produced a null message. Original message was: Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS
Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:

Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {}

6. For the enrichment topology, getting the below exception.
2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
request.required.acks = 1 was supplied but isn't a known config.
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
0.10.0.1
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
a7a17cdec9eaa6c5
2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
enrichmentErrorOutputBolt:(1)
2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
System classloader
2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
threatIntelSplitBolt:(14)
2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
enrichmentSplitBolt:(3)
2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
hostEnrichmentBolt:(5)
2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
enrichmentJoinBolt:(2)
2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
org.apache.metron.jackson.core.JsonParseException: Unrecognized token
'enrichments': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
at
org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)

~[stormjar.jar:?]
at
org.apache.metron.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)

~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3466)

~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2625)

~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:854)

~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)

~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)

~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)

~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)

~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
~[stormjar.jar:?]

7. For the Indexing topology , getting below exception
com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
at
com.fasterxml.jackson.core.metron.elasticsearch.JsonParser._constructError(JsonParser.java:1581)

~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)

~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)

~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)

~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)

~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:737)

~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)

~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)

~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)

~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
~[stormjar.jar:?]
at
org.apache.metron.common.configuration.IndexingConfigurations.updateSensorIndexingConfig(IndexingConfigurations.java:52)

~[stormjar.jar:?]


Kindly help to resolve the issue.

Regards
Girish N

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message