kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shyam P <shyamabigd...@gmail.com>
Subject Re: Does anyone fixed Producer TimeoutException problem ?
Date Wed, 03 Jul 2019 08:25:35 GMT
thank you.
Senthil ,
Why batch.size=65536  & retries=100, its big number right ? do we need this
much for streaming applications ?

Regards,
Shyam

On Wed, Jul 3, 2019 at 1:30 PM SenthilKumar K <senthilec566@gmail.com>
wrote:

> `*Partition = -1` - *This explains why are you getting timeout error.
>
> Why dont you use Default Partitioner ?:
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
>
> Try only with below producer properties :
> bootstrap.servers=<<>>>
> acks =1
> retries=100
> batch.size=65536
> linger.ms=10
> key.serializer=<<>>
> value.serializer=<<>>>
> compression.type=<<snappy>>
> buffer.memory=104857600
>
> --Senthil
>
>
> On Wed, Jul 3, 2019 at 1:18 PM Shyam P <shyamabigdata@gmail.com> wrote:
>
>> Hi SenthilKumar,
>> thanks a lot .
>>
>> Yeah I set up local set up and print the log with partition info which
>> shows as below
>>
>> 2019-07-03 02:48:28.822 INFO 7092 --- [lt-dispatcher-2]
>> c.s.c.p.p.CompanyInfoPartitioner : Topic : inbound_topic Key = 597736248-
>> Entropy Cayman Solar Ltd.-null-null-null *Partition = -1* 2019-07-03
>> 02:48:28.931 ERROR 7092 --- [ad | producer-1]
>> o.s.k.support.LoggingProducerListener : Exception thrown when sending a
>> message with key='597736248- Entropy Cayman Solar Ltd.-null-null-null' and
>> payload='com.spgmi.ca.prescore.model.Company@8b12343' to topic
>> inbound_topic :
>>
>> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
>> inbound_topic --1: 104 ms has passed since batch creation plus linger time
>>
>> My topics inbound_topic has two partitions as you see below
>> C:\Software\kafka\kafka_2.11-2.1.1\bin\windows>kafka-topics.bat --describe
>> --zookeeper localhost:2181 --topic inbound_topic Topic:inbound_topic
>> PartitionCount:2 ReplicationFactor:1 Configs: Topic: inbound_topic
>> Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: inbound_topic Partition: 1
>> Leader: 0 Replicas: 0 Isr: 0
>>
>> *But my producer seems to trying to send to Partition = -1.*
>>
>> My partition logic is as below
>>
>> int p = (((String)key).hashCode() * Integer.MAX_VALUE) % numPartitions;
>>         logger.info("Topic : "+ topic + "\t Key = " + (String)key + " Partition =
" + p );
>>
>> On key i am doing hashCode(). What need to be corrected here to avoid
>> this negative number partition number ? i.e. Partition = -1
>>
>> What should be my partition key logic like ?
>>
>> any help highly appreciated.
>> Regards,
>> Shyam
>>
>> On Tue, Jul 2, 2019 at 8:48 PM SenthilKumar K <senthilec566@gmail.com>
>> wrote:
>>
>>> Does it happen to all partitions or only few partitions ? Can you make
>>> sure your local setup working fine ? Were you able to produce using
>>> console-producer ?
>>>
>>> Example :
>>> EVERE: Expiring 7 record(s) for topic-9{partition:9}: 30022 ms has
>>> passed since last append
>>> Expiring 9 record(s) for topic-2{partition:2}: 30015 ms has passed
>>> since batch creation plus linger time
>>>
>>> --Senthil
>>>
>>> On Tue, Jul 2, 2019 at 5:34 PM Shyam P <shyamabigdata@gmail.com> wrote:
>>>
>>>> Thanks a lot Senthil for quick reply.
>>>> I am using  kafka_2.11-2.1.1 .
>>>> In your case  Kafka Producer Client in One DataCenter and Kafka Broker
>>>> in other DataCenter  but in my case I installed Kafka on the same machine
>>>> where Producer is running.
>>>> i.e. currently I am in development mode , so everything now on my local
>>>> for timebeing ...i.e. Kafka broker , zk and my producer code in eclipse.
>>>>
>>>> If is is a set up issue at least it should run fine in my local right.
>>>> I tried several producer configurations combinations as explained in
>>>> the SOF link.
>>>>
>>>> So not sure now what is the issue and how to fix it ?
>>>>
>>>> Is in your case the issue fixed ?
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>> On Tue, Jul 2, 2019 at 5:12 PM SenthilKumar K <senthilec566@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Shyam, We also faced `TimeoutException: Expiring 1 record(s)` issue
>>>>> in
>>>>> our Kafka Producer Client. As described here
>>>>> <
>>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has
>>>>> >
>>>>> ,
>>>>> first we tried increasing request timeout but that didn't help.  We had
>>>>> setup like Kafka Producer Client in One DataCenter and Kafka Broker in
>>>>> other DataCenter & thats why the producer failed to push records
to
>>>>> brokers
>>>>> on time due to network issue. In your case , Could be setup issue ?
>>>>>
>>>>> --Senthil
>>>>>
>>>>> On Tue, Jul 2, 2019 at 3:57 PM Shyam P <shyamabigdata@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >  I am facing the below issue.
>>>>> >
>>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)
>>>>> > for <topic-name> 229 ms has passed since batch creation plus
linger
>>>>> > time
>>>>> >
>>>>> >
>>>>> > I tried many producer configuration settings. more details below
:
>>>>> >
>>>>> >
>>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has
>>>>> >
>>>>> > But nothing working.
>>>>> >
>>>>> > Can anyone plz help me , what is wrong here and how to fix it ?
>>>>> >
>>>>> > thanks,
>>>>> > Shyam
>>>>> >
>>>>>
>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message