spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody Koeninger (JIRA)" <>
Subject [jira] [Commented] (SPARK-17938) Backpressure rate not adjusting
Date Sat, 15 Oct 2016 13:48:20 GMT


Cody Koeninger commented on SPARK-17938:

There was pretty extensive discussion of this on list, should link or summarize it.

Couple of things here:

 100 is the default minimum rate for pidestimator.  If you're willing to write code, put more
logging in to determine why that rate isn't being configured, or hardcode it to a different
number. I have successfully adjusted that rate using spark configuration.

The other thing is that if your system takes way longer than 1 second to process 100k records,
100k obviously isn't a reasonable max. Many large batches will be defined during the time
that first batch is running, before back pressure is involved at all. Try a lower max.

> Backpressure rate not adjusting
> -------------------------------
>                 Key: SPARK-17938
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: Samy Dindane
> spark-streaming 2.0.1 and spark-streaming-kafka-0-10 version is 2.0.1. Same behavior
with 2.0.0 though.
> is set to 30000
> spark.streaming.kafka.maxRatePerPartition is set to 100000
> spark.streaming.backpressure.enabled is set to true
> `batchDuration` of the streaming context is set to 1 second.
> I consume a Kafka topic using KafkaUtils.createDirectStream().
> My system can handle 100k records batches, but it'd take more than 1 seconds to process
them all. I'd thus expect the backpressure to reduce the number of records that would be fetched
in the next batch to keep the processing delay inferior to 1 second.
> Only this does not happen and the rate of the backpressure stays the same: stuck in `100.0`,
no matter how the other variables change (processing time, error, etc.).
> Here's a log showing how all these variables change but the chosen rate stays the same: (I would have attached a
file but I don't see how).
> Is this the expected behavior and I am missing something, or is this  a bug?
> I'll gladly help by providing more information or writing code if necessary.
> Thank you.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message