storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Kuzmenko <f1she...@gmail.com>
Subject Re: Kafka monitor unable to get offset lag
Date Wed, 01 Feb 2017 19:53:05 GMT
Yes running from command line gives me json with topic lag.
What is secured setup?

On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah <pshah@hortonworks.com> wrote:

> Hi Igor,
>
>
>
> Just to clarify, are you saying that running storm-kafka-monitor from
> command line with the arguments you pasted returns the spout lags correctly
> ? Also, can you confirm if you are running in a secured setup or not ?
>
>
>
> *From: *Igor Kuzmenko <f1sherox@gmail.com>
> *Reply-To: *"user@storm.apache.org" <user@storm.apache.org>
> *Date: *Wednesday, February 1, 2017 at 5:05 AM
> *To: *"user@storm.apache.org" <user@storm.apache.org>
> *Subject: *Re: Kafka monitor unable to get offset lag
>
>
>
> Found in logs with debug level:
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname:
> org.apache.storm.kafka.spout.KafkaSpout
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json
> configuration: {config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
> config.topics=gtp,, config.groupid=test, topology.tasks=1}
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run:
> [/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test,
> -b, dn-06.ru:6667,dn-10.ru:6667]
>
> 2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing
> failed, assuming message as error message: Unable to get offset lags for
> kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException:
> Timeout expired while fetching topic metadata
>
>
>
> Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t
> gtp -g test -b dn-06.ru:6667,dn-10.ru:6667  returns correct json with
> spout lags. There's no timeout command executing in a second.
>
>
>
>
>
>
>
> On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <spicoflorin@gmail.com>
> wrote:
>
> Hello!
>
>   You can check how your topic is consumed  and its health via the scripts:
>
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --list
>
>  ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> <your-broker>:6667 --describe --group <your-group>
>
>
>
> I hope it helps.
>
>  Florin
>
>
>
> On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1sherox@gmail.com> wrote:
>
> Yes, topology process data and works fine.
>
> I couldn't find any exceptions in storm logs.   access-web-ui.log contains
> only these lines
>
>
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
> principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
>
>
> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <pshah@hortonworks.com>
> wrote:
>
> Hi Igor,
>
>
>
> When you say topology is working fine do you mean you see data flowing?
> Can you try to look up the logs for ui server and paste relevant lines here
> if any?
>
>
>
> Priyank
>
> Sent from my iPhone
>
>
> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1sherox@gmail.com> wrote:
>
> I've launched topology with new kafka spout. Topology by it self working
> fine, but looking at storm UI I see kafka-monitor exception:
>
> *Unable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
> while fetching topic metadata*
>
>
>
> Maybe I forgot to configure something, but then how topology reads
> messages?
>
>
>
>
>
>
>

Mime
View raw message