kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Alberto.Fo...@natwestmarkets.com.INVALID>
Subject Per topic MBeans leaking
Date Fri, 13 Jan 2017 16:12:20 GMT
Hi,

In our application we create and delete topics dynamically. In our dev/test environment, where
we create and delete a lot of topics, I've noticed that the memory usage of the Kafka brokers
slowly increases and then noticed the huge amount of MBeans related to topics that should
have been deleted (e.g. kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d).

Also, deleting a topic often produces (what I think is) noise in the logs at WARN level, for
example:
2017-01-13 14:52:46 WARN  PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener
on 1]: Ignoring request to delete non-existing topics dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2

This behaviour is easy to reproduce with a basic Kafka cluster with two brokers, just create
and delete topics few times. Sometimes the MBeans for the topic are deleted and sometimes
not.

Maybe I'm deleting the topics in the wrong way? I'm using the AdminUtils class in the Java
API:
AdminUtils.deleteTopic(zkUtils, topicName);

Topics are created using the AdminUtils class as well:
AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, topicConfiguration,
kafka.admin.RackAwareMode.Enforced$.MODULE$);

The Kafka version is 0.10.0.1.

Thanks,
Alberto


******************************************************************

NatWest Markets is a marketing name of The Royal Bank of Scotland plc. 

This communication and any attachments are confidential and intended solely for the addressee.
If you are not the intended recipient please advise us immediately and delete it. Unless specifically
stated in the message or otherwise indicated, you may not duplicate, redistribute or forward
this message and any attachments are not intended for distribution to, or use by any person
or entity in any jurisdiction or country where such distribution or use would be contrary
to local law or regulation. The Royal Bank Of Scotland plc or any affiliated entity ("RBS")
accepts no responsibility for any changes made to this message after it was sent.

Unless otherwise specifically indicated, the contents of this communication and its attachments
are for information purposes only and should not be regarded as an offer or solicitation to
buy or sell a product or service, confirmation of any transaction, a valuation, indicative
price or an official statement. Where this communication has been prepared by an RBS trading
desk, that desk may have a position or interest in the products or services mentioned that
is inconsistent with any views expressed in this message. In evaluating the information contained
in this message, you should know that it could have been previously provided to other clients
and/or internal RBS personnel, who could have already acted on it.

RBS cannot provide absolute assurances that all electronic communications (sent or received)
are secure, error free, not corrupted, incomplete or virus free and/or that they will not
be lost, mis-delivered, destroyed, delayed or intercepted/decrypted by others. Therefore RBS
disclaims all liability with regards to electronic communications (and the contents therein)
if they are corrupted, lost destroyed, delayed, incomplete, mis-delivered, intercepted, decrypted
or otherwise misappropriated by others.

Any electronic communication that is conducted within or through RBS systems will be subject
to being archived, monitored and produced to regulators and in litigation in accordance with
RBS's policy and local laws, rules and regulations. Unless expressly prohibited by local law,
electronic communications may be archived in countries other than the country in which you
are located, and may be treated in accordance with the laws and regulations of the country
of each individual included in the entire chain.

Copyright 2014 The Royal Bank of Scotland plc. All rights reserved. See http://www.natwestmarkets.com/legal/s-t-discl.html
for further risk disclosure.

******************************************************************

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message