kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
Date Thu, 10 Dec 2015 06:50:01 GMT
John,

Your question was a bit confusing because CorrelationID has a specific
meaning in the Kafka protocols, but those are an implementation detail that
you, as a user of the API, should not need to worry about. CorrelationIDs
as defined by the protocol are not exposed to the user (and do not
correspond to individual Kafka messages -- they correspond to
request/response pairs in the protocol).

In Kafka you can uniquely identify a message by the (topic, partition,
offset) tuple. To match up the record passed to a producer and the record
received by a consumer, you would use, as you suggested in your first email:

Producer: RecordMetadata.topic(), RecordMetadata.partition(),
RecordMetadata.offset()
Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(),
ConsumerRecord.offset()

-Ewen

On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jmenke@numerix.com> wrote:

> Gwen, thanks for the reply:
>
> Just to confirm - are we talking about the combination of the IDs that I
> layed out as being the "CorrelationID" ?
>
> I guess the alternative is to pass our own CorrelationID as part of the
> data.   Would you agree with this?
>
> I do see something at this address that talks about a "user defined"
> CorrelationID
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
>
> CorrelationId:
>
> This is a user-supplied integer. It will be passed back in the response by
> the server, unmodified. It is useful for matching request and response
> between the client and server.
>
>
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gwen@confluent.io]
> Sent: Wednesday, December 09, 2015 12:50 PM
> To: users@kafka.apache.org
> Subject: Re: Match Producer and RecordMetadata with Consumer and
> ConsumerRecord
>
> Correlation ID is for a request (i.e. separate ID for produce request and
> a fetch request), not a record. So it can't be used in the way you are
> trying to.
>
> On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jmenke@numerix.com> wrote:
>
> > Can a correlationID be created from a ConsumerRecord that will allow
> > for identification of the corresponding RecordMetaData instance that
> > was returned from the Producer.send() method?
> >
> > I am Looking at the JavaDocs and the Producer returns RecordMetadata
> > which has the following signature:
> >
> > RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> > relativeOffset)
> >
> > I am not sure if this can match to
> >
> > ConsumerRecord(java.lang.String topic, int partition, long offset, K
> > key, V value)
> >
> > Can we match producer requests to Consumer Records by matching these
> > values?
> >
> > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
> >
> > In other words, can a CorrelationID be created from these values that
> > will allow Consumers to link back to the Producer send() event.  (In
> > the client calling code)
> >
> >
> >
> >
> >
> >
> >
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message