metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carolyn Duby <cd...@hortonworks.com>
Subject Re: taxi service doesn't respond
Date Tue, 04 Oct 2016 20:00:07 GMT
I changed the connection_config.json to reference node1 instead or localhost.  
I also restarted the opentaxii service.

I was able to get further in the load but then yarn, hdfs, and hbase started dying randomly.
 I think taxii oversubscribes the VM.

Thanks
Carolyn



On 10/4/16, 2:17 PM, "Carolyn Duby" <cduby@hortonworks.com> wrote:

>
>
>From: Carolyn Duby <cduby@hortonworks.com<mailto:cduby@hortonworks.com>>
>Date: Tuesday, October 4, 2016 at 2:15 PM
>To: Anand Subramanian <asubramanian@hortonworks.com<mailto:asubramanian@hortonworks.com>>,
Nicholas Allen <nallen@hortonworks.com<mailto:nallen@hortonworks.com>>
>Subject: taxi service doesn't respond
>
>
>I loaded up phishtank_com:
>
>
>[root@node1 ~]# service opentaxii status
>
>Checking opentaxii...                             Running
>
>guest.phishtank_com                                21779
>
>guest.Abuse_ch                                     0
>
>
>[root@node1 ~]# cat connection_config.json
>
>{
>
>  "endpoint" : "http://localhost:9000/services/discovery"
>
>  ,"username" : "guest"
>
>  ,"password" : "guest"
>
>  ,"type" : "DISCOVER"
>
>  ,"collection" : "guest.phishtank_com"
>
>  ,"table" : "threatintel"
>
>  ,"columnFamily" : "t"
>
>  ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
>
>}
>
>
>[root@node1 ~]# cat extractor.json
>
>{
>
>  "config": {
>
>    "columns": {
>
>      "ip": 0
>
>    },
>
>    "indicator_column": "ip",
>
>    "type" : "malicious_ip",
>
>    "separator" : ","
>
>  },
>
>  "extractor" : "STIX"
>
>}
>
>
>The Taxii load fails with the following error:
>
>
>[root@node1 ~]# /usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00"
-c ~/connection_config.json -e ~/extractor.json -p 10000
>
>WARNING: Use "yarn jar" to launch YARN applications.
>
>16/10/04 18:06:44 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery,
port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=domainname:FQDN,address:IPV_4_ADDR,
collection='guest.phishtank_com', subscriptionId='null', beginTime=Mon Aug 01 00:00:00 UTC
2016, table=threatintel:t}
>
>16/10/04 18:06:44 INFO taxii.TaxiiHandler: Initializing client..
>
>16/10/04 18:06:44 INFO taxii.TaxiiHandler: Discovering endpoint
>
>16/10/04 18:06:47 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest
=> org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse)
>
>16/10/04 18:06:47 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll
>
>16/10/04 18:06:48 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll
for guest.phishtank_com
>
>16/10/04 18:06:48 INFO taxii.TaxiiHandler: Polling...10/4/16 6:06 PM
>
>16/10/04 18:06:48 INFO taxii.TaxiiHandler: Begin Time: 2016-08-01T00:00:00Z
>
>16/10/04 18:07:11 ERROR taxii.TaxiiHandler: The target server failed to respond
>
>org.apache.metron.httpcore.dataload.NoHttpResponseException: The target server failed
to respond
>
>at(DefaultHttpResponseParser.java:143)
>
>at org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>
>at org.apache.metron.httpcore.dataload.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
>
>at org.apache.metron.httpcore.dataload.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>at java.lang.reflect.Method.invoke(Method.java:497)
>
>at org.apache.metron.httpcore.dataload.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138)
>
>at com.sun.proxy.$Proxy33.receiveResponseHeader(Unknown Source)
>
>at org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
>
>at org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
>
>at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
>
>at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>
>at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
>
>at java.util.TimerThread.mainLoop(Timer.java:555)
>
>at java.util.TimerThread.run(Timer.java:505)
>
>Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make request
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214)
>
>at java.util.TimerThread.mainLoop(Timer.java:555)
>
>at java.util.TimerThread.run(Timer.java:505)
>
>Caused by: org.apache.metron.httpcore.dataload.NoHttpResponseException: The target server
failed to respond
>
>at org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
>
>at org.apache.metron.httpcore.dataload.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>
>at org.apache.metron.httpcore.dataload.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
>
>at org.apache.metron.httpcore.dataload.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>at java.lang.reflect.Method.invoke(Method.java:497)
>
>at org.apache.metron.httpcore.dataload.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138)
>
>at com.sun.proxy.$Proxy33.receiveResponseHeader(Unknown Source)
>
>at org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
>
>at org.apache.metron.httpcore.dataload.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
>
>at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
>
>at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
>
>at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>
>at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
>
>at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
>
>... 2 more
>
Mime
View raw message