metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Miklavcic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies
Date Fri, 02 Mar 2018 18:16:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383908#comment-16383908
] 

Michael Miklavcic commented on METRON-1466:
-------------------------------------------

Nevermind, this isn't even getting to the indexing topology.

> The Elastic Search index are not getting generated in the latest metron deployments for
all the topologies
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: METRON-1466
>                 URL: https://issues.apache.org/jira/browse/METRON-1466
>             Project: Metron
>          Issue Type: Bug
>         Environment: Metron secured cluster 
>            Reporter: Mohan
>            Priority: Major
>         Attachments: bro_worker.log, enrichmnts_worker.log, indexing_worker.log
>
>
> The Elastic Search index are not getting generated in the latest metron deployments for
all the topologies , 
> I see that the data is getting published to the respective topics and also getting enriched
.
> below are the console consumer output for topics "bro" and "enrichments" I dont get anything
for "indexing" topic .
> {code:java}
> ./kafka-console-consumer.sh --zookeeper ctr-e138-1518143905142-61778-01-000002.hwx.site:2181
--security-protocol PLAINTEXTSASL --topic bro {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
./kafka-console-consumer.sh --zookeeper ctr-e138-1518143905142-61778-01-000002.hwx.site:2181
--security-protocol PLAINTEXTSASL --topic enrichments {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"]
trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center
PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"897e534c-ed6b-458c-9b2d-8749a68ed447","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"]
trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center
PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"6caaab54-e78d-4d31-9b2c-5c2bb25d609f","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"]
trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center
PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"a7475614-789a-4221-88cb-0734ef5fdc0a","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"]
trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center
PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"f05fa5de-5549-4293-85cf-006c66dba5d4","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"]
trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center
PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"116cb86a-2f41-48a8-8cc0-ee15d25d8db6","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET
CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message