kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siva Ram <sivaraman...@gmail.com>
Subject Kafka Streams throughput performance overtime
Date Tue, 24 Jul 2018 01:19:38 GMT
 Hi,

I have a stream application that performs rollup from 15mins to Hourly,
then Hourly to Daily.  The process needs to be continuously run 24 hours
and each 15 mins approx 12 million records gets posted (a JSON record per
message) into the input topic. There are 3 separate processors
corresponding to the above, where Hourly and Daily maintains the state.
So, in hourly 10million needs to be retained ever hour in the state and in
daily overall 10 million.  42GB of memory is allocated for the whole
application and throughput is fine for until first 10hrs, after that it
degrades significantly.  Any suggestions on this to identify the delay and
to increase the throughput would be of great help?  We are on Kafka 1.0.0

Thanks
Siva

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message