kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Krasser <krass...@googlemail.com>
Subject Atomic write of message batch to single partition
Date Fri, 24 Apr 2015 06:37:17 GMT

I'm using Kafka (in a Scala/Java project) and trying to find out 
how to atomically write n messages (message batch) to a single topic 
partition. Is there any client API that gives such a guarantee? I 
couldn't find a clear answer reading the documentation, API docs (of the 
old and new producer) and mailing list archives - sorry if I missed 
something obvious.

With the new org.apache.kafka.clients.producer.KafkaProducer, the send 
method has only a single ProducerRecord parameter and record 
accumulation and batch-sending of records is an implementation detail 
(which can be controlled to some extend by the batch.size configuration 
setting etc but not by user-defined message batches). So it seems that 
the new producer cannot be used for that. Are there any plans to support 
that in future versions?

Only the old kafka.producer.Producer allows me to pass a user-defined 
KeyedMessage batch to its send method. What are the semantics of this 
method when producer.type=sync and all KeyedMessages for a given send 
call are targeted at the same topic partition? Are these messages being 
written atomically to the partition?

Are there other options to achieve atomic writes of user-defined message 
batches (except making the batch a single Kafka message)?

Thanks for any hints!


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message