kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manikumar Reddy <ku...@nmsworks.co.in>
Subject Re: Kafka getMetadata api
Date Fri, 02 Jan 2015 17:34:12 GMT
Hi,

One option is to partition the data using key and consume from relevant
partition.
Or your current approach (filtering messages in the application) should be
OK.

Using separate getMetaData/getkey and getMessage may hit the consumer
performance/throughput.


Regards,
Kumar

On Fri, Jan 2, 2015 at 9:53 PM, Mukesh Jha <me.mukesh.jha@gmail.com> wrote:

> Any pointers guys?
> On 1 Jan 2015 15:26, "Mukesh Jha" <me.mukesh.jha@gmail.com> wrote:
>
> > Hello Experts,
> >
> > I'm using a kafka topic to store bunch of messages where the key contains
> > metadata and value is the data (avro file in our case).
> > There are multiple consumers for each topic and the consumer can decide
> if
> > the message is relevant for it or not based on the metadata i.e. the key
> of
> > the message.
> > Using this a group-consumer can check the key if the message is required
> > by it and then it can retrieve the entire message otherwise it'll just
> > commit the offset and move on to the next message.
> >
> > So I was wondering if kafka has an api in kafka that lets the consumer to
> > get just the metadata i.e. key, something like getMetadata instead
> > of getMessageAndMetadata?
> >
> > If kafka has something like this can you help me out with some
> > documentation for the same?
> > I think this will be useful in a lot of scenarios so if its not there I
> > can file a JIRA and take a dig at it. Let me know what you all think.
> >
> > Thanks for your help & suggestions.
> >
> > --
> > Thanks & Regards,
> >
> > *Mukesh Jha <me.mukesh.jha@gmail.com>*
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message