kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Swapnil Ghike <sgh...@linkedin.com>
Subject Re: Max message size & fetch size related?
Date Tue, 25 Sep 2012 00:18:37 GMT
Yes, that should work as long as your maxMessageSize limit is
appropriately set in producer and server.

Thanks,
Swapnil

On 9/24/12 4:22 PM, "Puneet Mehta" <mehta.puns@gmail.com> wrote:

>Hi Swapnil, 
>
>Thanks for the quick response.
>
>So if I am using a SimpleConsumer, I will need to create the fetch
>request like the following, right?
>
>// Fetch size as 5MB
>
>long fetchSize="5*1024*1024"
>FetchRequest fetchRequest = new FetchRequest("test", 0, offset,
>fetchSize);
>Thanks,
>
>Puneet Mehta
>
>
>
>On Monday, September 24, 2012 at 4:06 PM, Swapnil Ghike wrote:
>
>> Hi Puneet,
>> 
>> Yes you will need to bump up the maxMessageSize in server.KafkaConfig
>>and
>> fetchSize in consumer.ConsumerConfig.
>> 
>> server.KafkaConfig.maxMessageSize can be same as
>> producer.ProducerConfig.maxMessageSize.
>> 
>> 
>> You can set consumer.ConsumerConfig.fetchsize to a value greater than or
>> equal to producer.ProducerConfig.maxMessageSize.
>> 
>> Thanks,
>> Swapnil
>> 
>> On 9/24/12 3:51 PM, "Puneet Mehta" <mehta.puns@gmail.com
>>(mailto:mehta.puns@gmail.com)> wrote:
>> 
>> > Hi all,
>> > 
>> > We are using kafka 0.7.1
>> > 
>> > I am seeing this error in the producer ->
>> > 
>> > kafka.common.MessageSizeTooLargeException
>> > at
>> > 
>>kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyM
>>e
>> > ssageSize$1.apply(SyncProducer.scala:141)
>> > at
>> > 
>>kafka.producer.SyncProducer$$anonfun$kafka$producer$SyncProducer$$verifyM
>>e
>> > ssageSize$1.apply(SyncProducer.scala:139)
>> > at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>> > at kafka.message.MessageSet.foreach(MessageSet.scala:87)
>> > at
>> > 
>>kafka.producer.SyncProducer.kafka$producer$SyncProducer$$verifyMessageSiz
>>e
>> > (SyncProducer.scala:139)
>> > at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
>> > at
>> > 
>>kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.sc
>>a
>> > la:116)
>> > at 
>> > 
>>kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
>> > at 
>> > 
>>kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:102)
>> > at kafka.producer.ProducerPool.send(ProducerPool.scala:102)
>> > at kafka.producer.Producer.zkSend(Producer.scala:143)
>> > at kafka.producer.Producer.send(Producer.scala:105)
>> > at kafka.javaapi.producer.Producer.send(Producer.scala:104)
>> > 
>> > We are using the max message size as 1 <1000000>000000 <1000000>
>>bytes.
>> > 
>> > I am planning to bump this up to say 5 <5000000>000000 <5000000>
>>bytes.
>> > 
>> > I am just wondering do I need to change any other properties in
>> > producer/broker/consumer that may get impacted by this bump up?
>> > 
>> > Also, I came across this thread, which relates to fetch size relative
>>to
>> > max message size -> https://issues.apache.org/jira/browse/KAFKA-247
>> > 
>> > Could anyone of you advise me on places I may get impacted and change
>> > accordingly?
>> > 
>> > 
>> > Thanks,
>> > Puneet Mehta
>> > 
>> 
>> 
>> 
>
>


Mime
View raw message