kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID
Date Sun, 01 Dec 2019 20:53:47 GMT
Correct.

As long as there are only LOG statements about it, you don't need to
worry. If the producer cannot recover from this error internally, it
would throw an `UnknowdProducerIdException`; for this case, the write
was _not_ successful. Your application could handle this case by
creating a new `KafkaProducer` with the same `transactional.id` and
retry the transaction.

Note that KIP-360 is partly implemented in 2.4.0 already what should
reduce the likelihood to hit this issue.


-Matthias

On 12/1/19 9:42 AM, Jose Manuel Vega Monroy wrote:
> Hi Matthias,
> 
> So looking will be fixed in Kafka 2.5.0, as per https://issues.apache.org/jira/browse/KAFKA-8710,
so we will take into account to upgrade properly to that version when ready and stable.
> 
> We see this kind of errors in broker logs:
> 
> [2019-12-01 15:27:39,790] ERROR [ReplicaManager broker=3] Error processing append operation
on partition OB_SP11-12 (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.UnknownProducerIdException: Found no record of producerId=23001
on the broker at offset 24761in partition OB_SP11-12. It is possible that the last message
with the producerId=23001 has been removed due to hitting the retention limit.
> 
> Just to confirm, I guess they are correlated to producer errors right?
> 
> Thanks
> 
>  <http://www.williamhill.com/>
>  <http://www.whenthefunstops.co.uk/>
> Jose Manuel Vega Monroy
> Java Developer / Software Developer Engineer in Test
> Direct: +0035 0 2008038 (Ext. 8038)
> Email: jose.monroy@williamhill.com
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
> 
> 
> 
> 
> On 12/10/2019, 22:11, "Matthias J. Sax" <matthias@confluent.io> wrote:
> 
>     >> Does it mean producer’s metadata was removed due to retention time,
> 
>     Yes.
> 
>     >> and message is failed to be sent?
> 
>     No. Note, it's a warning not an error.
> 
>     The producer resets its internal state for this case and will send the
>     message successfully. It's a known issue and addressed via KIP-360 that
>     provides a more detailed explanation:
>     https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer
> 
> 
>     -Matthias
> 
>     On 10/12/19 12:18 PM, Jose Manuel Vega Monroy wrote:
>     > Hi there,
>     >
>     >
>     >
>     > We are seeing WARN log traces from time to time in our Kafka producer:
>     >
>     >
>     >
>     > 2019-10-12 15:19:14:707
>     > WARN[org.apache.kafka.clients.producer.internals.Sender] [Producer
>     > clientId=prdxpds21dsm001.prod.williamhill.plc] Got error produce
>     > response with correlation id 4899937 on topic-partition OB_SP25-3,
>     > retrying (0 attempts left). Error: UNKNOWN_PRODUCER_ID
>     >
>     >
>     >
>     > 2019-10-1220:02:21:686 WARN
>     > [org.apache.kafka.clients.producer.internals.Sender] [Producer
>     > clientId=prdxpds21dsm001.prod.williamhill.plc] Got errorproduce response
>     > with correlation id 5354758 on topic-partition OB_SP87-16, retrying (0
>     > attempts left). Error:UNKNOWN_PRODUCER_ID
>     >
>     >
>     >
>     > Does it mean producer’s metadata was removed due to retention time, and
>     > message is failed to be sent?
>     >
>     >
>     >
>     > I’m not sure but looking like happening since we enabled in producer:
>     >
>     >
>     >
>     > *enable.idempotence=true*
>     >
>     >
>     >
>     > Any idea?
>     >
>     >
>     >
>     > Thanks
>     >
>     >
>     >
>     > https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180
>     > <http://www.williamhill.com/>
>     >
>     > https://www.williamhillplc.com/content/signature/senet.gif?width=180
>     > <http://www.whenthefunstops.co.uk/>
>     >
>     >
>     >
>     > *Jose Manuel Vega Monroy **
>     > **Java Developer / Software Developer Engineer in Test*
>     >
>     > Direct: +*0035 0 2008038 (Ext. 8038)*
>     > Email: jose.monroy@williamhill.com <mailto:jose.monroy@williamhill.com>
>     >
>     > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>     >
>     >
>     >
>     >
>     >
>     >
>     >
> 
> 
> 
> Confidentiality: The contents of this e-mail and any attachments transmitted with it
are intended to be confidential to the intended recipient; and may be privileged or otherwise
protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate
or redistribute it by any means. Please delete it and any attachments and notify the sender
that you have received it in error. This e-mail is sent by a William Hill PLC group company.
The William Hill group companies include, among others, William Hill PLC (registered number
4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo
Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered
number C43260). Each of William Hill PLC and William Hill Organization Limited is registered
in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU,
UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office
at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United
States of America. WHG (International) Limited is registered in Gibraltar and has its registered
office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM
1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject
to contract; and are not an official statement, and do not necessarily represent the views,
of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William
Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for
any viruses contained within this e-mail and it is your responsibility to scan any emails
and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor
e-mail traffic data and also the content of e-mails for effective operation of the e-mail
system, or for security, purposes.
> 


Mime
View raw message