spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From allonsy <luke1...@gmail.com>
Subject spark.streaming.maxRatePerPartition parameter: what are the benefits?
Date Thu, 13 Aug 2015 14:50:29 GMT
Hello everyone,

in the new Kafka Direct API, what are the benefits of setting a value for
*spark.streaming.maxRatePerPartition*?

In my case, I have 2 seconds batches consuming ~15k tuples from a topic
split into 48 partitions (4 workers, 16 total cores).

Is there any particular value I should be setting the parameter to, in order
to achieve better performances? And what happens if I don't set the value at
all?

I could not find any detailed explanation about this.

Thank you!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-maxRatePerPartition-parameter-what-are-the-benefits-tp24241.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message