kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feroze Daud <khic...@yahoo.com.INVALID>
Subject log compaction scaling with ~100m messages
Date Wed, 07 Oct 2015 03:34:04 GMT
We have a use case where we want to store ~100m keys in kafka. Is there any problem with this
I have heard from some people using kafka, that kafka has a problem when doing log compaction
with those many number of keys.
Another topic might have around 10 different K/V pairs for each key in the primary topic.
The primary topic's keyspace is approx of 100m keys. We would like to store this in kafka
because we are doing a lot of stream processing on these messages, and want to avoid writing
another process to recompute data from snapshots.
So, in summary:
primary topic: ~100m keyssecondary topic: ~1B keys
Is it feasible to use log compaction at such a scale of data?
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message