kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avinash Herle <avinash.herl...@gmail.com>
Subject Re: Memory Leak in Kafka
Date Wed, 24 Jan 2018 17:18:50 GMT
Hi Ted,

I've posted this question on a kafka-user google group as well. Here is the
link <https://groups.google.com/forum/#!topic/kafka-clients/AeglVfsRCak>.
It has the attachments as well.

Thanks,
Avinash

On Tue, 23 Jan 2018 at 17:23 Ted Yu <yuzhihong@gmail.com> wrote:

> Did you attach two .png files ?
>
> Please use third party site since the attachment didn't come thru.
>
> On Tue, Jan 23, 2018 at 5:20 PM, Avinash Herle <avinash.herle04@gmail.com>
> wrote:
>
> >
> > Hi,
> >
> > I'm using Kafka as a messaging system in my data pipeline. I've a couple
> > of producer processes in my pipeline and Spark Streaming
> > <
> https://spark.apache.org/docs/2.2.1/streaming-kafka-0-10-integration.html>
> > and Druid's Kafka indexing service
> > <
> http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html
> >
> > as consumers of Kafka. The indexing service spawns 40 new indexing tasks
> > (Kafka consumers) every 15 mins.
> >
> > The heap memory used on Kafka seems fairly constant for an hour after
> > which it seems to shoot up to the max allocated space. The garbage
> > collection logs of Kafka seems to indicate a memory leak in Kafka. Find
> > attached the plots generated from the GC logs.
> >
> > *Kafka Deployment:*
> > 3 nodes, with 3 topics and 64 partitions per topic
> >
> > *Kafka Runtime jvm parameters:*
> > 8GB Heap Memory
> > 1GC swap Memory
> > Using G1GC
> > MaxGCPauseMilllis=20
> > InitiatingHeapOccupancyPercent=35
> >
> > *Kafka Versions Used:*
> > I've used Kafka version 0.10.0, 0.11.0.2 and 1.0.0 and find similar
> > behavior
> >
> > *Questions:*
> > 1) Is this a memory leak on the Kafka side or a misconfiguration of my
> > Kafka cluster?
> > 2) Druid creates new indexing tasks periodically. Does Kafka stably
> handle
> > large number of consumers being added periodically?
> > 3) As a knock on effect, We also notice kafka partitions going offline
> > periodically after some time with the following error:
> >     ERROR [ReplicaFetcherThread-18-2], Error for partition [topic1,2] to
> > broker
> 2:*org.apache.kafka.common.errors.UnknownTopicOrPartitionException*:
> > This server does not host this topic-partition. (kafka.server.
> > ReplicaFetcherThread)
> >
> > Can someone shed some light on the behavior being seen in my cluster?
> >
> > Please let me know if more details are needed to root cause the behavior
> > being seen.
> >
> > Thanks in advance.
> >
> > Avinash
> > [image: Screen Shot 2018-01-23 at 2.29.04 PM.png][image: Screen Shot
> > 2018-01-23 at 2.29.21 PM.png]
> >
> >
> >
> >
> > --
> >
> > Excuse brevity and typos. Sent from mobile device.
> >
> >
> > --
> >
> > Excuse brevity and typos. Sent from mobile device.
> >
>


-- 

Excuse brevity and typos. Sent from mobile device.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message