spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soumitra Siddharth Johri <soumitra.siddha...@gmail.com>
Subject Re: Spark Streaming : is spark.streaming.receiver.maxRate valid for DirectKafkaApproach
Date Tue, 10 May 2016 13:08:55 GMT
Also look at back pressure enabled. Both of these can be used to limit the rate

Sent from my iPhone

> On May 10, 2016, at 8:02 AM, chandan prakash <chandanbaranwal@gmail.com> wrote:
> 
> Hi,
> I am using Spark Streaming with Direct kafka approach.
> Want to limit number of event records coming in my batches.
> Have question regarding  following 2 parameters : 
> 1. spark.streaming.receiver.maxRate
> 2. spark.streaming.kafka.maxRatePerPartition
> 
> 
> The documentation (http://spark.apache.org/docs/latest/streaming-programming-guide.html#deploying-applications
) says .....
> " spark.streaming.receiver.maxRate for receivers and spark.streaming.kafka.maxRatePerPartition
for Direct Kafka approach "
> 
> Does it mean that  spark.streaming.receiver.maxRate  is valid only for Receiver based
approach only ?  (not the DirectKafkaApproach as well)
> 
> If yes, then how do we control total number of records/sec in DirectKafka ?.....because
spark.streaming.kafka.maxRatePerPartition  only controls max rate per partition and not whole
records    . There might be many partitions some with very fast rate and some with very slow
rate.
> 
> Regards,
> Chandan
> 
> 
> 
> -- 
> Chandan Prakash
> 

Mime
View raw message