kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukesh Jha <me.mukesh....@gmail.com>
Subject Re: Kafka getMetadata api
Date Sat, 03 Jan 2015 01:25:21 GMT
Indeed my message size varies b/w ~500kb to ~5mb per avro.

I am using kafka as a I need a scalable pub-sub messaging architecture with
multiple produces and consumers and guaranty of delivery.
Keeping data on filesystem or hdfs won't give me that.

Also In the link below [1] there is a linkedin's performance benchmark of
kafka wrt message size which shows that Kafka's throughput increases with
messages of size ~100kb+.

Agreed for kafka a record is key+value, I'm wondering if kafka can give us
a way to sneak peek a records metadata via its key.

[1]
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
On 3 Jan 2015 01:27, "Jayesh Thakrar" <j_thakrar@yahoo.com.invalid> wrote:

> Just wondering Mukesh - the reason you want this feature is because your
> value payload is not small (tens of kb). Don't know if that is the right
> usage of kafka. It might be worthwhile to store the avro files in a
> filesystem (regular, cluster fs, hdfs or even hbase) and the value in your
> kafka message can be the reference or uri for the avro file.
>
> That way you make the best use of each system's features and strengths.
>
> Kafka does have api to get metadata - the topics, partitions and  primary
> for partition etc. If we consider a key-value pair as a "record" than what
> you are looking for is to get a part of the record (ie key only) and not
> the whole record - so i would still consider that a data query/api.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message