kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Ollech <f.oll...@yieldlab.de>
Subject Producer garbage collection problem
Date Fri, 31 Jan 2014 10:26:52 GMT
Hi,

I am currently trying to setup Kafka 0.8 and ran into a problem with the
producer. Every time the server triggers a garbage collection, because
permgen space is full, the server descends into a very unusual memory state
where a lot of garbage collection is suddenly happening (~5-10% gc cpu
time, when normally it is 0-0.5%). Additionally a lot of objects suddenly
get promoted from eden space to survivor space, old gen and perm gen. This
obviously slows down the whole server and turns up the system load by a
lot.
This behaviour only appears when the producer is active during a perm gen
gc and can be remedied by reloading the producer(s). Since reloading the
producer(s) every 5 seconds really is not how it is supposed to work, I am
wondering if any one has any pointers as to where the problem may be?

My system has the following specs:
- oracle jdk 1.6.0_22-b04
- sending about 12000 messages per second, about 60 bytes each
- producer counts tried: 1, 10, 25, 50
- async producer
- ack makes no difference
- batch sizes tried: 100 - 5000

I made screenshot of the behaviour: http://i.imgur.com/xtlVNaK.png
at ~10:24:30 the full gc happens, surivor space fills up and old gen
becomes a sawtooth.
at ~10:34 I reloaded the producers and everything was fine again.

Cheers,
Flo

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message