kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Strąk <piotr.st...@relayr.io>
Subject Kafka Streams not fetching data from one partition
Date Thu, 11 Jul 2019 12:52:58 GMT
Hello,

I'm investigating an issue in which a Kafka Streams application does not 
consume from one of the partitions it was assigned. I'm using the 2.3.0 
version.

All the fetch requests are sent for two partitions only:

 > Using older server API v6 to send FETCH 
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=Topic_v2,partitions=[{partition=25,fetch_offset=71484113,log_start_offset=-1,partition_max_bytes=1048576},{partition=22,fetch_offset=71296374,log_start_offset=-1,partition_max_bytes=1048576}]}]}

with correlation id 1049090 to node 1

But there should be three of them; using kafka-consumer-groups.sh I can 
see that the third one (partition 23) that has no current offset.

 > Topic_v2    22         71296374        71296374 0 
Topic-c1f03-StreamThread-1-consumer
 > Topic_v2    23         -                      72830772
-          
Topic-c1f03-StreamThread-1-consumer
 > Topic_v2    25         71484113        71484113 0 
Topic-c1f03-StreamThread-1-consumer

It worked fine until the partition was revoked, but absolutely nothing 
happened afterwards. Notice that the (shortened by me) clientId has 
changed.

 > 2019-07-10 02:15:36.347 [INFO] 
ConsumerCoordinator:Topic-c1f03-StreamThread-1 - [Consumer 
clientId=Topic-c1f03-StreamThread-1-consumer, groupId=Topic] Setting 
offset for partition Topic_v2-23 to the committed offset 
FetchPosition{offset=68735327, offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=ip-10-0-1-81.us-west-2.compute.internal:9092 
(id: 3 rack: null), epoch=-1}}
 > 2019-07-10 02:15:36.383 [INFO] 
ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer 
clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Revoking 
previously assigned partitions [Topic_v2-23]
 > 2019-07-10 02:15:36.498 [INFO] 
ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer 
clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Setting 
newly assigned partitions: Topic_v2-23
 > 2019-07-10 02:15:36.506 [INFO] 
ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer 
clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Setting 
offset for partition Topic_v2-23 to the committed offset 
FetchPosition{offset=70030508, offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=ip-10-0-1-81.us-west-2.compute.internal:9092 
(id: 3 rack: null), epoch=-1}}

What could be a reason that this partition is not included in the fetch 
request? It uses a single thread to process it (num.stream.threads set 
to 1), so if the thread was locked it couldn't keep working on two other 
partitions, if I understand correctly.


Mime
View raw message