spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhailau, Alex" <Alex.Mikhai...@mlb.com>
Subject Re: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers were doubled
Date Wed, 04 Oct 2017 14:48:07 GMT
Filed SPARK-22200

From: "Mikhailau, Alex" <Alex.Mikhailau@mlb.com>
Date: Wednesday, October 4, 2017 at 10:43 AM
To: "user@spark.apache.org" <user@spark.apache.org>
Subject: Re: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers
were doubled

Just found the same exact issues in one of our long running clusters:

Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event of resharding
will not be dropped during deaggregation of Amazon Kinesis records.


Cannot find the shard given the shardId shardId-000000004454
With 2.2.0

-Alex

From: "Mikhailau, Alex" <Alex.Mikhailau@mlb.com>
Date: Wednesday, September 13, 2017 at 4:16 PM
To: "user@spark.apache.org" <user@spark.apache.org>
Subject: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers
were doubled

Has anyone seen the following warnings in the log after a kinesis stream has been re-sharded?

com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask

WARN Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event
of resharding will not be dropped during deaggregation of Amazon Kinesis records.


com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy

WARN Cannot find the shard given the shardId shardId-000000000599


Mime
View raw message