beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [beam] ameihm0912 commented on issue #10973: KinesisIO retry LimitExceededException
Date Wed, 26 Feb 2020 21:03:04 GMT
ameihm0912 commented on issue #10973: KinesisIO retry LimitExceededException
URL: https://github.com/apache/beam/pull/10973#issuecomment-591642429
 
 
   > Thank you for contribution! Have you seen this PR #9765 and a discussion over this
topic there? I think it tries to solve the similar problem, if I'm not mistaken.
   
   @aromanenko-dev thank you for the response.
   
   I had a quick look at the referenced PR and I suspect that may be solving a different issue,
that being transient errors associated with hitting read throughput rate limiting.
   
   Some additional context on the issue we are running into: in our case, we have a large
number of Kinesis streams in a single account. By default, Kinesis `DescribeStream` calls
are limited to 10 per second per account. So we easily run into this during graph construction
as we have a large number of streams that need to be setup in the same pipeline. This results
in a `RuntimeException` being thrown by the pipeline. So, our issue is happening during stream
initialization right off the bat, where as the referenced PR I believe is more to address
issues that are occurring while reading from an already initialized stream.
   
   It is possible I am misinterpreting something in the other PR however.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message