kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Rangwala <sumitrangw...@gmail.com>
Subject Re: Issue with topic deletion
Date Wed, 04 Feb 2015 00:11:57 GMT
On Tue, Feb 3, 2015 at 3:37 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> Hey Sumit,
>
> I thought you would be providing the actual steps to reproduce :)
>

I want to but some proprietary code prevents me to do it.


> Nevertheless, can you get all the relevant logs: state change logs and
> controller logs at the very least and if possible server logs and send
> those over?
>

Here are all the logs you requested (there are three brokers in my setup
k1, k2, k3): http://d.pr/f/1kprY/2quHBRRT (Gmail has issue with the file)


Sumit




>
> Joel
>
> On Tue, Feb 03, 2015 at 03:27:43PM -0800, Sumit Rangwala wrote:
> > In my setup kafka brokers are set for auto topic creation. In the
> scenario
> > below a node informs other nodes (currently 5 in total) about  a number
> of
> > new (non-existent) topics, and  all the nodes almost simultaneously open
> a
> > consumer for each of those topics. Sometime later another node informs
> all
> > other nodes of a new list of topics and each node, if they find that an
> > older topic exists in kafka, goes ahead and deletes the older topic.
> What I
> > have found is that many of the topics stay in the "marked for deletion"
> > state forever.
> >
> >
> > I get the list of topics using ZkUtils.getAllTopics(zkClient) and delete
> > topics using AdminUtils.deleteTopic(zkClient, topic). Since many nodes
> > might try to delete the same topic at the same time I do
> > see ZkNodeExistsException while deleting the topic, which I catch an
> > ignore. (e.g., org.apache.zookeeper.KeeperException$NodeExistsException:
> > KeeperErrorCode = NodeExists for
> > /admin/delete_topics/LAX1-GRIFFIN-r13-1423001701601)
> >
> > # State of one deleted topic on kafka brokers:
> > Topic:LAX1-GRIFFIN-r13-1423001701601 PartitionCount:8 ReplicationFactor:1
> > Configs:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 0 Leader: -1 Replicas:
> > 49558 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 1 Leader: -1 Replicas:
> > 49554 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 2 Leader: -1 Replicas:
> > 49557 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 3 Leader: -1 Replicas:
> > 49558 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 4 Leader: -1 Replicas:
> > 49554 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 5 Leader: -1 Replicas:
> > 49557 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 6 Leader: -1 Replicas:
> > 49558 Isr:
> > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 7 Leader: -1 Replicas:
> > 49554 Isr:
> >
> >
> > # Controller log says
> >
> > [2015-02-03 22:59:03,399] INFO [delete-topics-thread-49554], Deletion for
> > replicas 49557,49554,49558 for partition
> >
> [LAX1-GRIFFIN-r13-1423001701601,0],[LAX1-GRIFFIN-r13-1423001701601,6],[LAX1-GRIFFIN-r13-1423001701601,5],[LAX1-GRIFFIN-r13-1423001701601,3],[LAX1-GRIFFIN-r13-1423001701601,7],[LAX1-GRIFFIN-r13-1423001701601,1],[LAX1-GRIFFIN-r13-1423001701601,4],[LAX1-GRIFFIN-r13-1423001701601,2]
> > of topic LAX1-GRIFFIN-r13-1423001701601 in progress
> > (kafka.controller.TopicDeletionManager$DeleteTopicsThread)
> >
> > current time: Tue Feb  3 23:20:58 UTC 2015
> >
> >
> > Since I don't know the delete topic algorithm, I am not sure why sure
> these
> > topics are not garbage collected. I do have the complete setup running in
> > docker right now on my local box so please let me know if any more info
> is
> > required to troubleshoot this issue.
> >
> >
> > Furthermore, does ZkUtils.getAllTopics(zkClient) return "marked for
> > deletion" topic as well. If so, is there a easy way to get a list of
> active
> > topics (other than looking at all the topics in /admin/delete_topics/ and
> > taking a set difference with the topics returned
> > by ZkUtils.getAllTopics(zkClient) )
> >
> > Sumit
> > (More setup info below)
> >
> >
> > Setup
> > --------
> > Zookeeper: 3.4.6
> > Kafka broker: 0.8.2-beta
> > Kafka clients: 0.8.2-beta
> >
> > # Kafka boker settings (all other settings are default 0.8.2-beta
> settings)
> > kafka.controlled.shutdown.enable: 'FALSE'
> > kafka.auto.create.topics.enable: 'TRUE'
> > kafka.num.partitions: 8
> > kafka.default.replication.factor: 1
> > kafka.rebalance.backoff.ms: 3000
> > kafka.rebalance.max.retries: 10
> > kafka.log.retention.minutes: 1200
> > kafka.delete.topic.enable: 'TRUE'
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message