metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Elliston Ball <si...@simonellistonball.com>
Subject Re: Ambari Metrics Collector failing...
Date Thu, 07 Sep 2017 16:50:57 GMT
Correct, it’s not critical. Metrics can help a bit with debugging things like node hotspots
in Metron and in HDP generally, but it’s certainly not required to run. 

Simon


> On 7 Sep 2017, at 16:13, Zeolla@GMail.com <zeolla@gmail.com> wrote:
> 
> I wouldn't consider it a show stopper myself, happy to be persuaded
> otherwise.  I'm not even 100% sure it's related to Metron.  I just put it
> in Ambari's maintenance mode for now.
> 
> Jon
> 
> On Thu, Sep 7, 2017, 11:11 Laurens Vets <laurens@daemon.be> wrote:
> 
>> Is this something we need to fix for 0.4.1? Also, should I create  JIRA
>> ticket?
>> 
>> On 2017-09-06 16:45, Zeolla@GMail.com wrote:
>>> I'm seeing the same issue right now as well on my fresh bare metal
>>> install
>>> of HDP (no Metron yet), haven't dug into it further to troubleshoot.
>>> 
>>> Jon
>>> 
>>> On Wed, Sep 6, 2017, 18:22 Laurens Vets <laurens@daemon.be> wrote:
>>> 
>>>> In preparation of 0.4.1-rc, I'm trying to install the current github
>>>> master and I'm running into an issue with Ambari-metrics-collector.
>>>> "Metrics Collector" seems to start, but immediately turns red again
>>>> Ambari and stops.
>>>> 
>>>> Any idea what might be going on or where I can start troubleshooting
>>>> this?
>>>> 
>>>> In /var/log/hbase-ams-master-metron1.log I see lots of:
>>>> 
>>>> 2017-09-06 21:49:16,469 INFO  [HBase-Metrics2-1] impl.MetricsConfig:
>>>> loaded properties from hadoop-metrics2-hbase.properties
>>>> 2017-09-06 21:49:16,477 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics
>>>> sink.
>>>> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
>>>> serviceName = ams-hbase
>>>> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Collector Uri:
>>>> http://metron1:6188/ws/v1/timeline/metrics
>>>> 2017-09-06 21:49:16,491 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSinkAdapter: Sink timeline started
>>>> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSystemImpl:
>>>> Scheduled snapshot period at 10 second(s).
>>>> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSystemImpl:
>>>> HBase metrics system started
>>>> 2017-09-06 21:49:16,518 WARN  [HBase-Metrics2-1] lib.Interns: Metrics
>>>> intern cache overflow at 2011 for
>>>> MetricsSystem={MetricsSystem=MetricsInfo
>>>> Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem
>>>> record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem
>>>> record}}
>>>> 2017-09-06 21:49:17,564 WARN
>>>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
>>>> caught end of stream exception
>>>> EndOfStreamException: Unable to read additional data from client
>>>> sessionid 0x15e5929270d0001, likely client has closed socket
>>>>         at
>>>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>>>>         at
>>>> 
>>>> 
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>>>>         at java.lang.Thread.run(Thread.java:745)
>>>> 2017-09-06 21:49:17,565 INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
>>>> Closed socket connection for client /10.0.0.11:
>>>> 45058 which had sessionid 0x15e5929270d0001q
>>>> 
>>>> Additionally, I also see:
>>>> 
>>>> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics
>>>> sink.
>>>> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
>>>> serviceName = ams-hbase
>>>> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
>>>> timeline.HadoopTimelineMetricsSink: Collector Uri:
>>>> http://metron1:6188/ws/v1/timeline/metrics
>>>> 2017-09-06 22:19:16,470 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSinkAdapter: Sink timeline started
>>>> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSystemImpl:
>>>> Scheduled snapshot period at 10 second(s).
>>>> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1]
>>>> impl.MetricsSystemImpl:
>>>> HBase metrics system started
>>>> 2017-09-06 22:20:26,491 INFO  [timeline]
>>>> timeline.HadoopTimelineMetricsSink: Unable to connect to collector,
>>>> http://metron1:6188/ws/v1/timeline/metrics
>>>> This exceptions will be ignored for next 100 times
>>>> 
>>>> 2017-09-06 22:20:26,491 WARN  [timeline]
>>>> timeline.HadoopTimelineMetricsSink: Unable to send metrics to
>>>> collector
>>>> by address:http://metron1:6188/ws/v1/timeline/metrics
>>>> 
>>>> Which ok because metrics collector isn't working...
>>>> 
>>>> Any ideas?
>>>> 
>> 
> -- 
> 
> Jon


Mime
View raw message