kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From I PVP <i...@hotmail.com>
Subject What makes a message key mandatory and how to turn it off?
Date Wed, 04 May 2016 08:55:12 GMT
HI all,

What makes a message key mandatory and how to turn it off ?

I am migrating the messaging piece of a java application from activemq to kafka.
The application was publishing messages to kafka(0.9.0) with no issues while running on single
broker on my dev machine.
After turning it into multi-broker, by enabling 2 additional brokers, now I am seeing error
messages on all producers saying that "Error when sending message to topic <topic name>
with key: null". The error happens on producing messages from Java client and for kafka-console-producer.sh
too.

As a newbie I am not doing anything fancy around partitions, key and etc.

The topics were being create with the following: /opt/kafka/bin/kafka-topics.sh --zookeeper
localhost:2181 --topic <topic name> --create

After going multi-broker I  was forced to define partitions because the command above starting
saying "Missing required argument "[partitions]” , so the all  topics are being create with
the following:

/opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --topic <topic name> --create
 --replication-factor 2 -partitions 1


Messages were being sent by :
...
ProducerRecord<String, Object> producerRecord = new ProducerRecord<String, Object>(topicName,
message);
producer.send(producerRecord);
...

To make things working  again after the multi-broker change and the sunden mandatory message
Key requirement  I change the code to use a ramdonUUID as the message key.
...
ProducerRecord<String, Object> producerRecord = new ProducerRecord<String, Object>(topicName,UUID.randomUUID().toString(),
message);

producer.send(producerRecord);
...

After the change to multi-broker all topics were deleted, all log files were deleted , zookeeper
entries were cleaned and topics were recreated.

Even now working fine from the java client with the UUID as the message key  it does not work
from the command line,  it looks like from the command line there is no way to set the message
key and the command line is critical when needed to do quick tests.

If none logic at the applications consuming the messages requires a key why  is it forcing
to set a key ? The applications consuming the messages  only need to consume the message as
they come to the topic.

Is there a way to turn off the mandatory message key on a multi-broker situation?

Thanks

--
IPVP

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message