kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sameer Kumar <sam.kum.w...@gmail.com>
Subject Re: Strange rebalancing exception in Kafka 1.0.0
Date Thu, 14 Dec 2017 12:23:19 GMT
Hi All,

This is only reproducible when I have 3 nodes in my cluster even in the
start of the app, everything works fine on 2 nodes.

I have tried this again and faced the same error, this time I
increased MAX_PARTITION_FETCH_BYTES_CONFIG to 10MB from default 1MB, still
getting the same error.

Any thoughts ?



[2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to
rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39)
(kafka.coordinator.group.GroupCoordinator)
[2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group
c-7-aq32 generation 26 (__consumer_offsets-39)
(kafka.coordinator.group.GroupCoordinator)
[2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received
from leader for group c-7-aq32 for generation 26
(kafka.coordinator.group.GroupCoordinator)
*[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90]
Appending metadata message for group c-7-aq32 generation 26 failed due to
org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN
error code to the client (kafka.coordinator.group.GroupMetadataManager)*
[2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to
rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39)
(kafka.coordinator.group.GroupCoordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1
3-consumer-e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1
2-consumer-b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-8
-consumer-1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-2
-consumer-be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-5
-consumer-f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-4
-consumer-a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-5
-consumer-6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-2
-consumer-d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-5
-consumer-c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1
0-consumer-9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1
-consumer-10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-8
-consumer-6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1
4-consumer-58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-9
-consumer-f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-3
-consumer-7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1
2-consumer-e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1
4-consumer-f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-4
-consumer-f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)
[2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1
-consumer-fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has
failed, removing it from the group (kafka.coordinator.group.Group
Coordinator)

On Wed, Dec 13, 2017 at 11:43 AM, Sameer Kumar <sam.kum.work@gmail.com>
wrote:

> Hi All,
>
> Any pointers to the above issue that I can explore further.
>
> -Sameer.
>
> On Tue, Dec 12, 2017 at 5:40 PM, Sameer Kumar <sam.kum.work@gmail.com>
> wrote:
>
>> HI Ismael,
>>
>> This is what I see in the logs, I tried this twice and got the same
>> exception.
>>
>> This is only reproducible when I have 3 nodes in my cluster even in the
>> start of the app, everything works fine on 2 nodes.
>>
>> [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to
>> rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39)
>> (kafka.coordinator.group.GroupCoordinator)
>> [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group
>> c-7-aq32 generation 26 (__consumer_offsets-39)
>> (kafka.coordinator.group.GroupCoordinator)
>> [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received
>> from leader for group c-7-aq32 for generation 26
>> (kafka.coordinator.group.GroupCoordinator)
>> *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90]
>> Appending metadata message for group c-7-aq32 generation 26 failed due to
>> org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN
>> error code to the client (kafka.coordinator.group.GroupMetadataManager)*
>> [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to
>> rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39)
>> (kafka.coordinator.group.GroupCoordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 13-consumer-e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 12-consumer-b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-
>> 8-consumer-1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 2-consumer-be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-
>> 5-consumer-f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 4-consumer-a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 5-consumer-6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 2-consumer-d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 5-consumer-c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 10-consumer-9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 1-consumer-10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 8-consumer-6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 14-consumer-58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 9-consumer-f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-
>> 3-consumer-7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-
>> 12-consumer-e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-
>> 14-consumer-f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-
>> 4-consumer-f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
>> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-
>> 1-consumer-fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has
>> failed, removing it from the group (kafka.coordinator.group.Group
>> Coordinator)
>> [2017-
>>
>> On Tue, Dec 12, 2017 at 4:07 PM, Ismael Juma <ismael@juma.me.uk> wrote:
>>
>>> Can you please check the broker logs for errors?
>>>
>>> Ismael
>>>
>>> On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar <sam.kum.work@gmail.com>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > Facing an strange exception while running Kafka Streams. I am reading
>>> from
>>> > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0.
>>> >
>>> > Now, this error has started appearing recently. The application runs
>>> fine
>>> > on 2 nodes, but as soon as a 3rd node is added, it starts throwing
>>> > exception. Please find the stacktrace attached.
>>> >
>>> > 2017-12-12 15:32:55 ERROR Kafka010Base:47 - Exception caught in thread
>>> > c-7-aq32-5648256f-9142-49e2-98c0-da792e6da48e-StreamThread-5
>>> > org.apache.kafka.common.KafkaException: Unexpected error from
>>> SyncGroup:
>>> > The server experienced an unexpected error when processing the request
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator$SyncGroupResponseHandler.handle(Abstract
>>> Coordinator.
>>> > java:566)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator$SyncGroupResponseHandler.handle(Abstract
>>> Coordinator.
>>> > java:539)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator$CoordinatorResponseHandler.
>>> > onSuccess(AbstractCoordinator.java:808)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator$CoordinatorResponseHandler.
>>> > onSuccess(AbstractCoordinator.java:788)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > RequestFuture$1.onSuccess(RequestFuture.java:204)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > RequestFuture.fireSuccess(RequestFuture.java:167)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > RequestFuture.complete(RequestFuture.java:127)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(
>>> > ConsumerNetworkClient.java:506)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerNetworkClient.firePendingCompletedRequests(
>>> > ConsumerNetworkClient.java:353)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
>>> >         at org.apache.kafka.clients.consumer.internals.
>>> > ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
>>> >         at org.apache.kafka.clients.consumer.KafkaConsumer.
>>> > pollOnce(KafkaConsumer.java:1138)
>>> >         at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
>>> > KafkaConsumer.java:1103)
>>> >         at org.apache.kafka.streams.processor.internals.
>>> > StreamThread.pollRequests(StreamThread.java:851)
>>> >         at org.apache.kafka.streams.processor.internals.
>>> > StreamThread.runOnce(StreamThread.java:808)
>>> >         at org.apache.kafka.streams.processor.internals.
>>> > StreamThread.runLoop(StreamThread.java:774)
>>> >         at org.apache.kafka.streams.processor.internals.
>>> > StreamThread.run(StreamThread.java:744)
>>> >
>>> >
>>> >
>>> > -Sameer.
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message