kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ari Flink <flinks...@gmail.com>
Subject Re: unable to delete topic with 0.8.2 rc2
Date Mon, 26 Jan 2015 18:27:01 GMT
unsubscribe

On Mon, Jan 26, 2015 at 8:14 AM, Harsha <kafka@harsha.io> wrote:

> Jun,
>       I made an attempt at fixing that issue as part of this JIRA
>       https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha <kafka@harsha.io> wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > log.OffsetIndex - Deleting index
> > > > /mypath/mytopic-0/00000000000000000000.index
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> log.LogManager
> > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > > > [Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion for replicas 7 for partition
> > > > [mytopic,0] of topic mytopic in progress
> > > > 2015-01-26 07:02:14,316  INFO [Controller-6-to-broker-7-send-thread]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > > > [Topic=mytopic,Partition=0,Replica=7]
> > > > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to NonExistentReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7]
> > > > 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> > > > controller.PartitionStateMachine - [Partition state machine on
> > > > Controller 6]: Invoking state change to OfflinePartition for
> > > > partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> > > > controller.PartitionStateMachine - [Partition state machine on
> > > > Controller 6]: Invoking state change to NonExistentPartition for
> > > > partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,325  WARN [kafka-request-handler-2]
> > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request
> > > > with correlation id 2619442 from client ReplicaFetcherThread-0-6 on
> > > > partition [mytopic,0] failed due to Partition [mytopic,0] doesn't
> > > > exist on 6
> > > > 2015-01-26 07:02:14,371  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic successfully
> > > > completed
> > > > 2015-01-26 07:02:14,376  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$TopicChangeListener -
> > > > [TopicChangeListener on Controller 6]: New topics: [Set()], deleted
> > > > topics: [Set()], new partition replica assignment [Map()]
> > > > 2015-01-26 07:02:14,403  WARN [request-expiration-task]
> > > > server.ReplicaManager - [Replica Manager on Broker 6]: Fetch request
> > > > with correlation id 2054626 from client
> > > > myconsumer-1415333962444-cc78e67e-0-6 on partition [mytopic,0] failed
> > > > due to Partition [mytopic,0] doesn't exist on 6
> > > > 2015-01-26 07:02:14,446  INFO [kafka-request-handler-5]
> > > > admin.AdminUtils$ - Topic creation
> > > > {"version":1,"partitions":{"0":[7,6]}}
> > > > 2015-01-26 07:02:14,457  INFO [kafka-request-handler-5]
> > > > server.KafkaApis - [KafkaApi-6] Auto creation of topic mytopic with 1
> > > > partitions and replication factor 2 is successful!
> > > > 2015-01-26 07:02:14,458  INFO [kafka-network-thread-27330-2]
> > > > network.Processor - Closing socket connection to /10.1.10.4.
> > > > 2015-01-26 07:02:14,466  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$TopicChangeListener -
> > > > [TopicChangeListener on Controller 6]: New topics: [Set(mytopic)],
> > > > deleted topics: [Set()], new partition replica assignment
> > > > [Map([mytopic,0] -> List(7, 6))]
> > > > 2015-01-26 07:02:14,466  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.KafkaController - [Controller 6]: New topic creation
> > > > callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,466  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.KafkaController - [Controller 6]: New partition creation
> > > > callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,466  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine - [Partition state machine on
> > > > Controller 6]: Invoking state change to NewPartition for partitions
> > > > [mytopic,0]
> > > > 2015-01-26 07:02:14,467  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to NewReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,468  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine - [Partition state machine on
> > > > Controller 6]: Invoking state change to OnlinePartition for
> partitions
> > > > [mytopic,0]
> > > > 2015-01-26 07:02:14,530  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OnlineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,532  INFO [kafka-request-handler-5] log.Log -
> > > > Completed load of log mytopic-0 with log end offset 0
> > > > 2015-01-26 07:02:14,533  INFO [kafka-request-handler-5]
> log.LogManager
> > > > - Created log for partition [mytopic,0] in /mypath with properties
> > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000,
> > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
> > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096,
> > > > retention.bytes -> 50000000000, min.insync.replicas -> 1,
> > > > cleanup.policy -> delete, unclean.leader.election.enable -> true,
> > > > segment.ms -> 604800000, max.message.bytes -> 1000012,
> flush.messages
> > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5,
> retention.ms
> > > > -> 86400000, segment.jitter.ms -> 0}.
> > > > 2015-01-26 07:02:14,540  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,540  INFO [kafka-request-handler-5] log.Log -
> > > > Truncating log mytopic-0 to offset 0.
> > > > 2015-01-26 07:02:14,547  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Added fetcher for partitions List([[mytopic,0], initOffset 0 to
> broker
> > > > id:7,host:mybroker:54321] )
> > > >
> > > > Here’s the log in the other broker with a replica of the topic.
> > > >
> > > > 2015-01-26 07:02:14,312  INFO [kafka-request-handler-7]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-2]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-2]
> > > > log.OffsetIndex - Deleting index
> > > > /mypath/mytopic-0/00000000000000000000.index
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-2]
> log.LogManager
> > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > > 2015-01-26 07:02:14,530  INFO [kafka-request-handler-0]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 7]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,532  INFO [kafka-request-handler-0] log.Log -
> > > > Completed load of log mytopic-0 with log end offset 0
> > > > 2015-01-26 07:02:14,533  INFO [kafka-request-handler-0]
> log.LogManager
> > > > - Created log for partition [mytopic,0] in /mypath with properties
> > > > {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000,
> > > > segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
> > > > delete.retention.ms -> 86400000, index.interval.bytes -> 4096,
> > > > retention.bytes -> 50000000000, min.insync.replicas -> 1,
> > > > cleanup.policy -> delete, unclean.leader.election.enable -> true,
> > > > segment.ms -> 604800000, max.message.bytes -> 1000012,
> flush.messages
> > > > -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5,
> retention.ms
> > > > -> 86400000, segment.jitter.ms -> 0}.
> > > >
> > > > On Mon, Jan 26, 2015 at 1:26 AM, Jun Rao <jun@confluent.io> wrote:
> > > >
> > > > Could you do a get on /admin/deleted_topics and see if its cversion
> > > > changes
> > > > > after issuing the delete topic command? This will tell us whether
> the
> > > > > path /admin/deleted_topics//mytopic
> > > > > is actually created or not. Anything in the controller log that
> > > indicates a
> > > > > delete topic event is fired?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg <jbr@squareup.com
> >
> > > wrote:
> > > > >
> > > > > > yes
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao <jun@confluent.io>
> wrote:
> > > > > >
> > > > > > > Do you have delete.topic.enable turned on in all brokers?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg <
> jbr@squareup.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > So far, I have been unable to get delete topic to
work, with
> > > release
> > > > > > > > candidate 2 for 0.8.2.
> > > > > > > >
> > > > > > > > It worked ok when I ran it in the debugger locally,
on a
> single
> > > node
> > > > > > > > instance. But when I run it in our staging environment,
it
> is not
> > > > > > > > successfully even marking the topic for delete, for
some
> reason.
> > > > > > > >
> > > > > > > > I am setting delete.topic.enable to true (and see
the
> > > confirmation of
> > > > > > > this
> > > > > > > > in the startup logs, e.g.):
> > > > > > > >
> > > > > > > > INFO [main] utils.VerifiableProperties - Property
> > > delete.topic.enable
> > > > > > > > is overridden to true
> > > > > > > >
> > > > > > > > I run this command:
> > > > > > > >
> > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > > > > > myzkconnect:12345/mynamespace --delete --topic mytopic
> > > > > > > >
> > > > > > > > log4j:WARN No appenders could be found for logger
> > > > > > > > (org.I0Itec.zkclient.ZkConnection).
> > > > > > > > log4j:WARN Please initialize the log4j system properly.
> > > > > > > > log4j:WARN No appenders could be found for logger
> > > > > > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > > > > > log4j:WARN See
> > > http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > > > > > for more info.
> > > > > > > > log4j:WARN Please initialize the log4j system properly.
> > > > > > > > log4j:WARN See
> > > http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > > > > > for more info.
> > > > > > > > Topic mytopic is marked for deletion.
> > > > > > > > Note: This will have no impact if delete.topic.enable
is not
> set
> > > to
> > > > > > true.
> > > > > > > >
> > > > > > > > I then do a —list which should at least show the
topic
> marked for
> > > > > > > deletion:
> > > > > > > >
> > > > > > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > > > > > myzkconnect:12345/mynamespace --list --topic mytopic
> > > > > > > >
> > > > > > > > log4j:WARN No appenders could be found for logger
> > > > > > > > (org.I0Itec.zkclient.ZkConnection).
> > > > > > > > log4j:WARN No appenders could be found for logger
> > > > > > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > > > > > log4j:WARN Please initialize the log4j system properly.
> > > > > > > > log4j:WARN Please initialize the log4j system properly.
> > > > > > > > log4j:WARN See
> > > http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > > > > > for more info.
> > > > > > > > log4j:WARN See
> > > http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > > > > > for more info.
> > > > > > > > mytopic
> > > > > > > >
> > > > > > > > Note, it doesn’t list it as ‘marked for deletion’.
> Furthermore,
> > > > > waiting
> > > > > > > > multiple hours still doesn’t result in the topic
being
> deleted.
> > > > > > > >
> > > > > > > > The topic has 1 partition, 2 replicas, and no data
stored.
> > > > > > > >
> > > > > > > > In zookeeper, the /admin/deleted_topics/ path is empty.
> > > > > > > >
> > > > > > > > The zookeeper code looks pretty straightforward, but
for some
> > > reason
> > > > > is
> > > > > > > not
> > > > > > > > writing the deleted_topics path.  We are running zookeeper
> 3.4.6.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Jason
> > > > > > > > ​
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > ​
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message