kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Alberto.Fo...@natwestmarkets.com.INVALID>
Subject RE: Per topic MBeans leaking
Date Fri, 13 Jan 2017 17:02:32 GMT
Sure, done: KAFKA-4629

Thanks,
Alberto


-----Original Message-----
From: Guozhang Wang [mailto:wangguoz@gmail.com] 
Sent: 13 January 2017 16:32
To: users@kafka.apache.org
Subject: Re: Per topic MBeans leaking

*********************************************
" This message originates from outside our organisation. Consider carefully whether you should
click on any links, open any attachments or reply. If in doubt, forward to ~ Phishing"
*********************************************

The way you used admin tools to create and delete topics looks reasonable to me. I suspect
there may be some race conditions causing the metrics to not be de-registered.

Could you file a JIRA to track this issue so we can investigate?


Guozhang

On Fri, Jan 13, 2017 at 8:12 AM, <Alberto.Forti@natwestmarkets.com.invalid>
wrote:

> Hi,
>
> In our application we create and delete topics dynamically. In our 
> dev/test environment, where we create and delete a lot of topics, I've 
> noticed that the memory usage of the Kafka brokers slowly increases 
> and then noticed the huge amount of MBeans related to topics that 
> should have been deleted (e.g. 
> kafka.server:type=BrokerTopicMetrics,name=
> TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d).
>
> Also, deleting a topic often produces (what I think is) noise in the 
> logs at WARN level, for example:
> 2017-01-13 14:52:46 WARN  
> PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener 
> on 1]: Ignoring request to delete non-existing topics
> dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2
>
> This behaviour is easy to reproduce with a basic Kafka cluster with 
> two brokers, just create and delete topics few times. Sometimes the 
> MBeans for the topic are deleted and sometimes not.
>
> Maybe I'm deleting the topics in the wrong way? I'm using the 
> AdminUtils class in the Java API:
> AdminUtils.deleteTopic(zkUtils, topicName);
>
> Topics are created using the AdminUtils class as well:
> AdminUtils.createTopic(zkUtils, topicName, partitions, 
> replicationFactor, topicConfiguration, 
> kafka.admin.RackAwareMode.Enforced$.MODULE$);
>
> The Kafka version is 0.10.0.1.
>
> Thanks,
> Alberto
>
>
> ******************************************************************
>
> NatWest Markets is a marketing name of The Royal Bank of Scotland plc.
>
> This communication and any attachments are confidential and intended 
> solely for the addressee. If you are not the intended recipient please 
> advise us immediately and delete it. Unless specifically stated in the 
> message or otherwise indicated, you may not duplicate, redistribute or 
> forward this message and any attachments are not intended for 
> distribution to, or use by any person or entity in any jurisdiction or 
> country where such distribution or use would be contrary to local law 
> or regulation. The Royal Bank Of Scotland plc or any affiliated entity 
> ("RBS") accepts no responsibility for any changes made to this message after it was sent.
>
> Unless otherwise specifically indicated, the contents of this 
> communication and its attachments are for information purposes only 
> and should not be regarded as an offer or solicitation to buy or sell 
> a product or service, confirmation of any transaction, a valuation, 
> indicative price or an official statement. Where this communication 
> has been prepared by an RBS trading desk, that desk may have a 
> position or interest in the products or services mentioned that is 
> inconsistent with any views expressed in this message. In evaluating 
> the information contained in this message, you should know that it 
> could have been previously provided to other clients and/or internal RBS personnel, who
could have already acted on it.
>
> RBS cannot provide absolute assurances that all electronic 
> communications (sent or received) are secure, error free, not 
> corrupted, incomplete or virus free and/or that they will not be lost, 
> mis-delivered, destroyed, delayed or intercepted/decrypted by others. 
> Therefore RBS disclaims all liability with regards to electronic 
> communications (and the contents
> therein) if they are corrupted, lost destroyed, delayed, incomplete, 
> mis-delivered, intercepted, decrypted or otherwise misappropriated by 
> others.
>
> Any electronic communication that is conducted within or through RBS 
> systems will be subject to being archived, monitored and produced to 
> regulators and in litigation in accordance with RBS's policy and local 
> laws, rules and regulations. Unless expressly prohibited by local law, 
> electronic communications may be archived in countries other than the 
> country in which you are located, and may be treated in accordance 
> with the laws and regulations of the country of each individual 
> included in the entire chain.
>
> Copyright 2014 The Royal Bank of Scotland plc. All rights reserved. 
> See http://www.natwestmarkets.com/legal/s-t-discl.html for further 
> risk disclosure.
>
> ******************************************************************
>



--
-- Guozhang
******************************************************************

NatWest Markets is a marketing name of The Royal Bank of Scotland plc. 

This communication and any attachments are confidential and intended solely for the addressee.
If you are not the intended recipient please advise us immediately and delete it. Unless specifically
stated in the message or otherwise indicated, you may not duplicate, redistribute or forward
this message and any attachments are not intended for distribution to, or use by any person
or entity in any jurisdiction or country where such distribution or use would be contrary
to local law or regulation. The Royal Bank Of Scotland plc or any affiliated entity ("RBS")
accepts no responsibility for any changes made to this message after it was sent.

Unless otherwise specifically indicated, the contents of this communication and its attachments
are for information purposes only and should not be regarded as an offer or solicitation to
buy or sell a product or service, confirmation of any transaction, a valuation, indicative
price or an official statement. Where this communication has been prepared by an RBS trading
desk, that desk may have a position or interest in the products or services mentioned that
is inconsistent with any views expressed in this message. In evaluating the information contained
in this message, you should know that it could have been previously provided to other clients
and/or internal RBS personnel, who could have already acted on it.

RBS cannot provide absolute assurances that all electronic communications (sent or received)
are secure, error free, not corrupted, incomplete or virus free and/or that they will not
be lost, mis-delivered, destroyed, delayed or intercepted/decrypted by others. Therefore RBS
disclaims all liability with regards to electronic communications (and the contents therein)
if they are corrupted, lost destroyed, delayed, incomplete, mis-delivered, intercepted, decrypted
or otherwise misappropriated by others.

Any electronic communication that is conducted within or through RBS systems will be subject
to being archived, monitored and produced to regulators and in litigation in accordance with
RBS's policy and local laws, rules and regulations. Unless expressly prohibited by local law,
electronic communications may be archived in countries other than the country in which you
are located, and may be treated in accordance with the laws and regulations of the country
of each individual included in the entire chain.

Copyright 2014 The Royal Bank of Scotland plc. All rights reserved. See http://www.natwestmarkets.com/legal/s-t-discl.html
for further risk disclosure.

******************************************************************
Mime
View raw message