kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject Kafka Streams app error while rebalancing
Date Tue, 05 Dec 2017 17:35:49 GMT
Hello,

We noticed that a kafka streams app is stuck in rebalance state with below
error.
Two instance of the app were running fine until a rebalace was
triggered(possibly due to network issue).
Both app instance are running(no app restart)
App itself doesn't create/use state store. NUM_STREAM_THREADS_CONFIG=2

I did see several tickets with similar errors that are marked as fixed. I'm
using version 0.10.2.1.

17/12/04 18:34:57 WARN StreamThread: Could not create task 0_2. Will retry.
org.apache.kafka.streams.errors.LockException: task [0_2] Failed to lock
the state directory for task 0_2
  at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:100)
  at
org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
  at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
  at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:864)
  at
org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1237)
  at
org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1210)
  at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:967)
  at
org.apache.kafka.streams.processor.internals.StreamThread.access$600(StreamThread.java:69)
  at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:234)
  at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:259)
  at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:352)
  at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
  at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
  at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1029)
  at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
  at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:592)
  at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
17/12/04 18:34:58 WARN StreamThread: Could not create task 0_8. Will retry.
org.apache.kafka.streams.errors.LockException: task [0_8] Failed to lock
the state directory for task 0_8


kafka-consumer-groups --new-consumer --bootstrap-server <...> --describe
--group GeoTest
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).

Warning: Consumer group 'GeoTest' is rebalancing.

I keep seeing the above lock exception continuously and app is not making
any progress. Any idea why it is stuck?
I read a few suggestions that required me to manually delete state
directory. I'd like to avoid that.

Thanks,
Srikanth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message