kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: console producer question
Date Wed, 09 Apr 2014 23:44:43 GMT
This may be because the 0.8 producer sticks to a partition during
metadata refresh intervals.  You can get around that by specifying a
key:

--property parse.key=true --property key.separator=###

Each line would then be:
KEY###MESSAGE

The key is used for partitioning but will also be stored with the
message but that shouldn't break your consumers if they extract just
the message. We have a partition key which is thrown away after
partitioning but that has not been integrated with the console
producer.

Thanks,

Joel

On Wed, Apr 09, 2014 at 07:13:16PM -0400, Ian Friedman wrote:
> Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1. One of the
tools in the 0.7 distribution was something called Producer Shell, which we used in some cron
jobs to do some manual addition of messages (for messages that got dropped for various reasons).
So in 0.8 that is gone, and I have tried replacing it with the console Producer. It seems
like the console producer only produces to a single partition whereas the producer shell used
to randomly distribute the messages. Is this behavior configurable? AM I missing something
obvious? 
> 
> Thanks in advance
> --Ian


Mime
View raw message