kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brice Dutheil <brice.duth...@gmail.com>
Subject Broker Health check
Date Mon, 11 Jan 2016 16:55:59 GMT

I and my team are beginning with kafka so I/we may not have read everything
or understood everything yet.

The context is we’d like to bootstrap kafka in our own codebase, and first
step is to have a deployable walking skeleton that the team can grow along
our experience and needs.

We are starting with kafka, and surprisingly there’s no status
command for the broker, so we decided to do a comprehensive check but we
are still a bit confused on how to write a healthcheck that is not fragile.

Currently we would like to produce a message (e.g. a timestamp) on some
healthcheck topic, and expect a consumer to read this message, a grep would
check that the message has the same value.
However we are facing some behaviour that we don’t yet quite understand.

# first emit the message
timestamp=$(date -u "%Y-%m-%dT%H:%M%SZ)"echo $timestamp |
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
# try to consume the same message
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server
localhost:9092 --topic healthcheck --max-messages 1 --from-beginning |
grep $timestamp

[ 0 == $? ] && echo "kafka onfline" || echo "kafka offline"

In this case the consumer does not always *find* the matching message. We
have tried different options there ; with or without --max-messages,
--from-beginning, --timeout-ms, it does’nt behave as we thought by reading
the documentation. Sometime the message is read some time it isn’t, but we
don’t get why.

To be noted, I don’t understand how it is possible to read the latest

> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yat --partitions 1 --replication-factor
Created topic "yat".

> echo bob | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092

> echo charles | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
> echo yohan | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
> echo carlos | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
# Running the simple command just outputs nothing, and I need to kill
the process with ctrl-c, note the messages count is off by 1, or I'm
missing something
> bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic
^CProcessed a total of 1 messages
# Same with `--from-beginning`, the pocess don't finish up, and prints
everything since the beginning (that is to be expected).
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server
localhost:9092 --topic yat --from-beginning
^CProcessed a total of 8 messages
# With `--from-beginning` and `--max-messages` the consumer stops at
the very first message
[root@3af0ccad3182 kafka]# bin/kafka-console-consumer.sh
--new-consumer --bootstrap-server localhost:9092 --topic yat
--from-beginning --max-messages 1
Processed a total of 1 messages
# With `--max-messages` the latest message doesn't appear either and I
have to kill the consumer too
> bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic
yat --max-messages 1
^CProcessed a total of 1 messages

Also note the same happen with the old consumer with --zookeeper option.

Is there some limitation with the console consumer regarding offsets ? Is
there something I’m missing.

Should we write our own healthcheck ?

Thanks in advance for any feedback, pointers, etc.

— Brice

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message