kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "REYMOND Jean-max (BPCE-IT - SYNCHRONE TECHNOLOGIES)" <Jean-max.REYMOND.prestata...@bpce-it.fr>
Subject RE: Too many open files in kafka 0.9
Date Thu, 07 Dec 2017 09:20:13 GMT
According to 
https://issues.apache.org/jira/browse/KAFKA-3806

I have adjusted offset.retention.minutes and it seems that it solves my issue

-----Message d'origine-----
De : Ted Yu [mailto:yuzhihong@gmail.com] 
Envoyé : mercredi 29 novembre 2017 19:41
À : users@kafka.apache.org
Objet : Re: Too many open files in kafka 0.9

There is KAFKA-3317 which is still open.

Have you seen this ?

http://search-hadoop.com/m/Kafka/uyzND1KvOlt1p5UcE?subj=Re+Brokers+is+down+by+java+io+IOException+Too+many+open+files+

On Wed, Nov 29, 2017 at 8:55 AM, REYMOND Jean-max (BPCE-IT - SYNCHRONE
TECHNOLOGIES) <Jean-max.REYMOND.prestataire@bpce-it.fr> wrote:

> We have a cluster with 3 brokers and kafka 0.9.0.1. One week ago, we 
> decide to adjust log.retention.hours from 10 days to 2 days. Stop and 
> go the cluster and it is ok. But for one broker, we have every day 
> more and more datas and two days later crash with message too many 
> open files. lsof return 7400 opened files. We adjust to 10000 and 
> crash again. So, in our data repository, we remove all the datas and 
> run again and after a few minutes, cluster is OK. But now, after atfer 
> 6 hours, the two valid brokers have 72 GB and the other broker has 90 
> GB. lsof -p xxx returns 1030 and it is growing continously. I am sure 
> that tomorrow morning, we will have a crash.
>
> In the server.log of the broken broker,
>
> [2017-11-29 17:28:51,360] INFO Rolled new log segment for 
> '__consumer_offsets-27' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:31:28,836] INFO Rolled new log segment for 
> '__consumer_offsets-8' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:35:22,100] INFO Rolled new log segment for 
> '__consumer_offsets-12' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:37:55,984] INFO Rolled new log segment for 
> '__consumer_offsets-11' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:38:30,600] INFO [Group Metadata Manager on Broker 2]:
> Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.
> GroupMetadataManager)
> [2017-11-29 17:39:55,836] INFO Rolled new log segment for 
> '__consumer_offsets-16' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:43:38,300] INFO Rolled new log segment for 
> '__consumer_offsets-48' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:44:21,110] INFO Rolled new log segment for 
> '__consumer_offsets-36' in 1 ms. (kafka.log.Log)
> [2017-11-29 17:48:30,600] INFO [Group Metadata Manager on Broker 2]:
> Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.
> GroupMetadataManager)
>
> And in the same time on a valid broker
>
> [2017-11-29 17:44:46,704] INFO Deleting index 
> /pfic/kafka/data/kafka_data/__consumer_offsets-48/
> 00000000002686063378.index.deleted (kafka.log.OffsetIndex)
> [2017-11-29 17:44:47,341] INFO Deleting segment 2687254936 from log 
> __consumer_offsets-48. (kafka.log.Log)
> [2017-11-29 17:44:47,376] INFO Deleting index 
> /pfic/kafka/data/kafka_data/__consumer_offsets-48/
> 00000000002687254936.index.deleted (kafka.log.OffsetIndex)
> [2017-11-29 17:45:32,991] INFO Deleting segment 0 from log 
> __consumer_offsets-36. (kafka.log.Log)
> [2017-11-29 17:45:32,991] INFO Deleting segment 1769617973 from log 
> __consumer_offsets-36. (kafka.log.Log)
> [2017-11-29 17:45:32,993] INFO Deleting index 
> /pfic/kafka/data/kafka_data/__consumer_offsets-36/
> 00000000000000000000.index.deleted (kafka.log.OffsetIndex)
> [2017-11-29 17:45:32,993] INFO Deleting index 
> /pfic/kafka/data/kafka_data/__consumer_offsets-36/
> 00000000001769617973.index.deleted (kafka.log.OffsetIndex)
> [2017-11-29 17:45:33,593] INFO Deleting segment 1770704579 from log 
> __consumer_offsets-36. (kafka.log.Log)
> [2017-11-29 17:45:33,627] INFO Deleting index 
> /pfic/kafka/data/kafka_data/__consumer_offsets-36/
> 00000000001770704579.index.deleted (kafka.log.OffsetIndex)
> [2017-11-29 17:45:58,394] INFO [Group Metadata Manager on Broker 0]:
> Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.
> GroupMetadataManager)
>
> So, the broken broker never delete a segment. Of course, the three 
> brokers have the same configuration.
> What's happen ?
> Thanks for your advices,
>
>
> Jean-Max REYMOND
> BPCE Infogérance & Technologies
>
> ------------------------------------------------------------
> ------------------
> L'intégrité de ce message n'étant pas assurée sur Internet, BPCE-IT ne 
> peut être tenu responsable de son contenu. Si vous n'êtes pas 
> destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.
> The integrity of this message cannot be guaranteed on the Internet.
> BPCE-IT cannot therefore be considered responsible for the contents. 
> If you are not the intended recipient of this message, then please 
> delete it and notify the sender.
> ------------------------------------------------------------
> ------------------
>
------------------------------------------------------------------------------
L'intégrité de ce message n'étant pas assurée sur Internet, BPCE-IT ne peut être tenu
responsable de son contenu. Si vous n'êtes pas destinataire de ce message, merci de le détruire
et d'avertir l'expéditeur.
The integrity of this message cannot be guaranteed on the Internet. BPCE-IT cannot therefore
be considered responsible for the contents. If you are not the intended recipient of this
message, then please delete it and notify the sender.
------------------------------------------------------------------------------
Mime
View raw message