kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gustafson <ja...@confluent.io>
Subject Re: Coordinator issues with 0.9 consumer
Date Wed, 18 Nov 2015 18:53:50 GMT
Hi Martin,

Thanks for reporting this problem. I think maybe we're just not doing a
very good job of handling auto-commit errors internally and they end up
spilling into user logs. I added a JIRA to address this issue:
https://issues.apache.org/jira/browse/KAFKA-2860.

-Jason

On Wed, Nov 18, 2015 at 10:16 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hello Martin,
>
> Could you paste the consumer config values in this thread as well? And is
> the consumer co-located with the broker?
>
> Guozhang
>
> On Wed, Nov 18, 2015 at 7:40 AM, Martin Skøtt <
> martin.skoett@falconsocial.com> wrote:
>
> > Hi,
> >
> > I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new
> > API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my
> > own machine using the settings files in config/ and no authentication.
> > Build is done using Oracle JDK 8. I have 13 topics, each created with a
> > replication factor of 1.
> >
> > The first time my consumer connects to the newly created topics (before
> > anything has been written to the topic) it emits the following errors:
> >
> > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version :
> > 0.9.0.0
> > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId
> :
> > fe855f98263cb7b8
> > [main] ERROR
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto
> > offset commit failed.
> > org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The
> > group coordinator is not available.
> > [main] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> > - Marking the coordinator 2147483647 dead.
> > [main] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator
> > information, retrying.
> > [main] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> > - Marking the coordinator 2147483647 dead.
> > [main] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator
> > information, retrying.
> >
> > If I restart my consumer then it starts without issue.
> >
> > When I start writing to the topics then my consumer reads data, but will
> > often log messages like these:
> >
> > [main] ERROR
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto
> > offset commit failed.
> > org.apache.kafka.clients.consumer.internals.SendFailedException
> >
> > And occasionally these:
> > [main] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> > - Attempt to heart beat failed since member id is not valid, reset it and
> > try to re-join group.
> > [main] ERROR
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error
> > ILLEGAL_GENERATION occurred while committing offsets for group
> > audience-indexer-reader
> > [main] ERROR
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto
> > offset commit failed.
> > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be
> > completed due to group rebalance
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350)
> > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:157)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:384)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:236)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:209)
> > at
> >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:304)
> > at
> >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:861)
> > at
> >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829)
> >
> > Does anyone have any input on what might be happening?
> >
> > --
> > Martin Skøtt
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message