kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Objelean <alex.objel...@gmail.com>
Subject Re: Monitoring connection with kafka client
Date Fri, 17 Oct 2014 08:08:10 GMT
@Neha Narkede

Though monitoring the health of Kafka & Zookeeper clusters directly is
useful, it might not be enough.
Consider the following scenario:

You have a client consuming messages. Zookeeper and kafka nodes are
healthy. The monitor says everything is ok. For some reason, a connection
between client and zookeeper  fails (new firewall rule or connectivity
issues between different data centers, doesn't really matter). Since the
zookeeper and kafka nodes are still healthy, the monitor is still reporting
OK. The client retries to reconnect indefinitely, but unless a human is
watching the logs, the problem is not visible by anybody.

Having a listener which is invoked when the connection state is changed is
something that is provided by other messaging frameworks. Having this
feature added to kafka would be a nice addition.

Thanks,
Alex

On 17 October 2014 10:55, Alex Objelean <alex.objelean@gmail.com> wrote:

> @Otis  thanks for your answer.
>
> Of course it is not about manually watching connection. We already have
> monitoring tools in place. But I would expect the kafka client to provide
> means for get notified about potential connectivity issues.
> Actual behavior is this: the client retries to reconnect indefinitely (I
> see errors in logs). What I need, is a kind of listener which would be
> invoked when the connection state change. In other words, once the client
> notice the problem and is trying to reconnect, our monitoring tools would
> reflect the connection state as soon as the problem appears.
>
> Thanks,
> Alex
>
>
>
> On 16 October 2014 13:09, Alex Objelean <alex.objelean@gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to monitor the kafka connection on the consumer side. In other
>> words, if the broker cluster is unavailable (or zookeer dies), I would like
>> to know about that problem as soon as possible.
>> Unfortunately, I didn't find anything useful to achieve that when using
>> kafka library.
>> Are there any suggestions about how to fix this issue?
>>
>> Thanks,
>> Alex
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message