spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Nadler (JIRA)" <>
Subject [jira] [Created] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams
Date Mon, 12 Sep 2016 19:21:20 GMT
Jeff Nadler created SPARK-17510:

             Summary: Set Streaming MaxRate Independently For Multiple Streams
                 Key: SPARK-17510
             Project: Spark
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 2.0.0
            Reporter: Jeff Nadler

We use multiple DStreams coming from different Kafka topics in a Streaming application.

Some settings like maxrate and backpressure enabled/disabled would be better passed as config
to KafkaUtils.createStream and KafkaUtils.createDirectStream, instead of setting them in SparkConf.

Being able to set a different maxrate for different streams is an important requirement for
us; we currently work-around the problem by using one receiver-based stream and one direct

We would like to be able to turn on backpressure for only one of the streams as well.   

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message