kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: Very long consumer rebalances
Date Thu, 09 Aug 2018 09:32:33 GMT
In the simplest way, how have you implemented your consumer?

1) Does your consumers join a designated group, process messages, and then
closes all connection? Or does it stay open perpetually until server
shutdown?
2) Have you configured the session timeouts for client and zookeeper
accordingly?

Regards,

On 9 August 2018 at 08:00, Shantanu Deshmukh <shantanu88d@gmail.com> wrote:

>  I am facing too many problems these days. Now one of our consumer groups
> is rebalancing every now and then. And rebalance takes very low, more than
> 5-10 minutes. Even after re-balancing I see that only half of the consumers
> are active/receive assignment. Its all going haywire.
>
> I am seeing these logs in kafka consumer logs. Can anyone help me
> understand what is going on here? It is a very long piece of log, but
> someone please help me. I am desperately looking for any solution since
> more than 2 months now. But to no avail.
>
> [2018-08-09 11:39:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> group bulk-email-consumer failed: The request timed out.
> [2018-08-09 11:40:04] :: INFO  :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:04] :: WARN  :: ConsumerCoordinator$4:535 - Auto offset
> commit failed for group bulk-email-consumer: Offset commit failed with a
> retriable exception. You should retry committing offsets.
> [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.117:9092 (id: 1 rack: null)
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> group bulk-email-consumer failed: The request timed out.
> [2018-08-09 11:40:04] :: INFO  :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:04] :: WARN  :: ConsumerCoordinator$4:535 - Auto offset
> commit failed for group bulk-email-consumer: Offset commit failed with a
> retriable exception. You should retry committing offsets.
> [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.118:9092 (id: 2 rack: null)
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795004158,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=389784,client_id=consumer-26},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> sendTimeMs=1533795004157),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:04] :: INFO  ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795004718,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=363859,client_id=consumer-25},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> sendTimeMs=1533795004145),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:04] :: INFO  ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:05] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:08] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:11] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:13] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:13] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:14] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:17] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:20] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:21] :: INFO  :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-6] for group bulk-email-consumer
> [2018-08-09 11:40:21] :: INFO  :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> 869377f9ead1,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:22] :: INFO  :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-9] for group bulk-email-consumer
> [2018-08-09 11:40:22] :: INFO  :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:23] :: INFO  :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-8] for group bulk-email-consumer
> [2018-08-09 11:40:23] :: INFO  :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:26] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:26] :: DEBUG :: Metadata  :241 - Updated cluster metadata
> version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:28] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:28] :: DEBUG :: Metadata  :241 - Updated cluster metadata
> version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
> null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:34] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:34] :: DEBUG :: Metadata  :241 - Updated cluster metadata
> version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> metadata=''},
> bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: WARN  :: ConsumerCoordinator:554 - Auto offset
> commit failed for group bulk-email-consumer: Commit cannot be completed
> since the group has already rebalanced and assigned the partitions to
> another member. This means that the time between subsequent calls to poll()
> was longer than the configured max.poll.interval.ms, which typically
> implies that the poll loop is spending too much time message processing.
> You can address this either by increasing the session timeout or by
> reducing the maximum size of batches returned in poll() with
> max.poll.records.
> [2018-08-09 11:40:50] :: INFO  :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
> for group bulk-email-consumer
> [2018-08-09 11:40:50] :: INFO  :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=,protocol_type=consumer,group_protocols=
> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:50] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 0
> [2018-08-09 11:40:50] :: DEBUG ::
> ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> LEAVE_GROUP request ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> request=RequestSend(header={api_key=13,api_version=0,
> correlation_id=349313,client_id=consumer-19},
> body={group_id=bulk-email-consumer,member_id=consumer-
> 19-34903180-9cb3-496d-a913-b782476d9a52}),
> createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation id
> 349313 due to node 2147483647 being disconnected
> [2018-08-09 11:40:50] :: INFO  :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG ::
> ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> JOIN_GROUP request ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> request=RequestSend(header={api_key=11,api_version=1,
> correlation_id=349314,client_id=consumer-19},
> body={group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=300000,member_id=,protocol_
> type=consumer,group_protocols=[{protocol_name=range,
> protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> correlation id 349314 due to node 2147483647 being disconnected
> [2018-08-09 11:40:50] :: INFO  :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.118:9092 (id: 2 rack: null)
> [2018-08-09 11:40:51] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 2
> [2018-08-09 11:40:51] :: DEBUG :: Metadata  :241 - Updated cluster metadata
> version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795051063,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=349316,client_id=consumer-19},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> sendTimeMs=1533795051061),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:51] :: INFO  ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:51] :: INFO  :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=,protocol_type=consumer,group_protocols=
> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:17] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:17] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:01] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:34] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:37] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:40] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:43] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:49] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:52] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:55] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:58] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:01] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:13] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:25] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> assignment for group bulk-email-consumer using strategy range with
> subscriptions
> {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> Subscription(topics=[bulk-email]),
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> Subscription(topics=[bulk-email]),
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> Subscription(topics=[bulk-email]),
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> Subscription(topics=[bulk-email])}
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> assignment for group bulk-email-consumer:
> {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> Assignment(partitions=[bulk-email-3,
> bulk-email-4, bulk-email-5]),
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> Assignment(partitions=[bulk-email-0,
> bulk-email-1, bulk-email-2]),
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> Assignment(partitions=[bulk-email-6,
> bulk-email-7]),
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> Assignment(partitions=[bulk-email-8,
> bulk-email-9])}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending leader
> SyncGroup for group bulk-email-consumer to coordinator 10.189.179.116:9092
> (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> assignment=[{member_id=consumer-23-bdc073b6-1f90-
> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=38
> cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=38
> cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=34
> cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=34 cap=34]}]}
> [2018-08-09 11:43:26] :: INFO  :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO  :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO  :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO  :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO  :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-7, bulk-email-6] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO  :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO  :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO  :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-8, bulk-email-9] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-7, bulk-email-6]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-0, bulk-email-2, bulk-email-1]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-4, bulk-email-3, bulk-email-5]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-8, bulk-email-9]
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-7 to the committed offset 25435998
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-0 to the committed offset 25390842
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-6 to the committed offset 25463566
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-4 to the committed offset 25409183
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-8 to the committed offset 25465113
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-2 to the committed offset 25368327
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :665 - Ignoring fetched records
> for partition bulk-email-8 since it is no longer fetchable
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-3 to the committed offset 25422008
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-9 to the committed offset 25444488
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-1 to the committed offset 25369263
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :221 - Resetting offset for
> partition bulk-email-5 to the committed offset 25456518
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :470 - Ignoring fetched records
> for bulk-email-0 at offset 25390442 since the current position is 25390842
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :665 - Ignoring fetched records
> for partition bulk-email-6 since it is no longer fetchable
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher   :671 - Discarding stale fetch
> response for partition bulk-email-1 since its offset 25369219 does not
> match the expected offset 25369263
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}}
> for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25436018 for partition bulk-email-7
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25409183 for partition bulk-email-4
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25422008 for partition bulk-email-3
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25456530 for partition bulk-email-5
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25390842 for partition bulk-email-0
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25368339 for partition bulk-email-2
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25369263 for partition bulk-email-1
> [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> metadata=''},
> bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
> bulk-email-consumer
>
> On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <steve.cs.tian@gmail.com>
> wrote:
>
> > It's a very good and important doc so I think you should read it all.
> You
> > should get some idea from sections like *Detecting Consumer Failures* and
> > *Multi-threaded Processing* for your case.
> >
> > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <shantanu88d@gmail.com>
> > wrote:
> >
> > > Hi Steve,
> > >
> > > Could you please shed more light on this? What section should I
> revisit?
> > I
> > > am using high-level consumer. So I am simply calling consumer.close()
> > when
> > > I am shutting down the process. Is there any other method to be called
> > > before calling close()?
> > >
> > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <steve.cs.tian@gmail.com>
> > wrote:
> > >
> > > > Please re-read the javadoc of KafkaConsumer, make sure you know how
> to
> > > > wakeup/close consumer properly while shutting down your application.
> > Try
> > > > to understand the motivation of KIP-62 and adjust related timeout.
> > > >
> > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hklohar@gmail.com> wrote:
> > > >
> > > > > Try reducing below timer
> > > > > metadata.max.age.ms = 300000
> > > > >
> > > > >
> > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > shantanu88d@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java
> > app
> > > > > which
> > > > > > spawns many consumer threads consuming from different topics. For
> > > every
> > > > > > topic we have specified different consumer-group. A lot of times
> I
> > > see
> > > > > that
> > > > > > whenever this application is restarted a CG on one or two topics
> > > takes
> > > > > more
> > > > > > than 5 minutes to receive partition assignment. Till that time
> > > > consumers
> > > > > > for that topic don't consumer anything. If I go to Kafka broker
> and
> > > run
> > > > > > consumer-groups.sh and describe that particular CG I see that it
> is
> > > > > > rebalancing. There is time critical data stored in that topic and
> > we
> > > > > cannot
> > > > > > tolerate such long delays. What can be the reason for such long
> > > > > rebalances.
> > > > > >
> > > > > > Here's our consumer config
> > > > > >
> > > > > >
> > > > > > auto.commit.interval.ms = 3000
> > > > > > auto.offset.reset = latest
> > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > > check.crcs = true
> > > > > > client.id =
> > > > > > connections.max.idle.ms = 540000
> > > > > > enable.auto.commit = true
> > > > > > exclude.internal.topics = true
> > > > > > fetch.max.bytes = 52428800
> > > > > > fetch.max.wait.ms = 500
> > > > > > fetch.min.bytes = 1
> > > > > > group.id = otp-notifications-consumer
> > > > > > heartbeat.interval.ms = 3000
> > > > > > interceptor.classes = null
> > > > > > key.deserializer = class
> > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > max.partition.fetch.bytes = 1048576
> > > > > > max.poll.interval.ms = 300000
> > > > > > max.poll.records = 50
> > > > > > metadata.max.age.ms = 300000
> > > > > > metric.reporters = []
> > > > > > metrics.num.samples = 2
> > > > > > metrics.sample.window.ms = 30000
> > > > > > partition.assignment.strategy = [class
> > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > receive.buffer.bytes = 65536
> > > > > > reconnect.backoff.ms = 50
> > > > > > request.timeout.ms = 305000
> > > > > > retry.backoff.ms = 100
> > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > sasl.kerberos.service.name = null
> > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > sasl.mechanism = GSSAPI
> > > > > > security.protocol = SSL
> > > > > > send.buffer.bytes = 131072
> > > > > > session.timeout.ms = 300000
> > > > > > ssl.cipher.suites = null
> > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > ssl.key.password = null
> > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > ssl.keystore.location = null
> > > > > > ssl.keystore.password = null
> > > > > > ssl.keystore.type = JKS
> > > > > > ssl.protocol = TLS
> > > > > > ssl.provider = null
> > > > > > ssl.secure.random.implementation = null
> > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > ssl.truststore.password = [hidden]
> > > > > > ssl.truststore.type = JKS
> > > > > > value.deserializer = class
> > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > >
> > > > > > Please help.
> > > > > >
> > > > > > *Thanks & Regards,*
> > > > > > *Shantanu Deshmukh*
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message