kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Bejeck <b...@confluent.io>
Subject Re: Kafka Streams not fetching data from one partition
Date Thu, 11 Jul 2019 20:18:18 GMT
Hi Piotr,

Thanks for reporting this issue.  Can you provide full kafka-streams and
broker logs around the timeframe you observed this?

-Bill

On Thu, Jul 11, 2019 at 8:53 AM Piotr StrÄ…k <piotr.strak@relayr.io> wrote:

> Hello,
>
> I'm investigating an issue in which a Kafka Streams application does not
> consume from one of the partitions it was assigned. I'm using the 2.3.0
> version.
>
> All the fetch requests are sent for two partitions only:
>
>  > Using older server API v6 to send FETCH
> {replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=Topic_v2,partitions=[{partition=25,fetch_offset=71484113,log_start_offset=-1,partition_max_bytes=1048576},{partition=22,fetch_offset=71296374,log_start_offset=-1,partition_max_bytes=1048576}]}]}
>
> with correlation id 1049090 to node 1
>
> But there should be three of them; using kafka-consumer-groups.sh I can
> see that the third one (partition 23) that has no current offset.
>
>  > Topic_v2    22         71296374        71296374 0
> Topic-c1f03-StreamThread-1-consumer
>  > Topic_v2    23         -                      72830772 -
> Topic-c1f03-StreamThread-1-consumer
>  > Topic_v2    25         71484113        71484113 0
> Topic-c1f03-StreamThread-1-consumer
>
> It worked fine until the partition was revoked, but absolutely nothing
> happened afterwards. Notice that the (shortened by me) clientId has
> changed.
>
>  > 2019-07-10 02:15:36.347 [INFO]
> ConsumerCoordinator:Topic-c1f03-StreamThread-1 - [Consumer
> clientId=Topic-c1f03-StreamThread-1-consumer, groupId=Topic] Setting
> offset for partition Topic_v2-23 to the committed offset
> FetchPosition{offset=68735327, offsetEpoch=Optional.empty,
> currentLeader=LeaderAndEpoch{leader=ip-10-0-1-81.us-west-2.compute.internal:9092
>
> (id: 3 rack: null), epoch=-1}}
>  > 2019-07-10 02:15:36.383 [INFO]
> ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer
> clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Revoking
> previously assigned partitions [Topic_v2-23]
>  > 2019-07-10 02:15:36.498 [INFO]
> ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer
> clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Setting
> newly assigned partitions: Topic_v2-23
>  > 2019-07-10 02:15:36.506 [INFO]
> ConsumerCoordinator:Topic-3e717-StreamThread-1 - [Consumer
> clientId=Topic-3e717-StreamThread-1-consumer, groupId=Topic] Setting
> offset for partition Topic_v2-23 to the committed offset
> FetchPosition{offset=70030508, offsetEpoch=Optional.empty,
> currentLeader=LeaderAndEpoch{leader=ip-10-0-1-81.us-west-2.compute.internal:9092
>
> (id: 3 rack: null), epoch=-1}}
>
> What could be a reason that this partition is not included in the fetch
> request? It uses a single thread to process it (num.stream.threads set
> to 1), so if the thread was locked it couldn't keep working on two other
> partitions, if I understand correctly.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message