spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: spark streaming with kafka source, how many concurrent jobs?
Date Sat, 11 Mar 2017 02:24:39 GMT
That config I not safe. Please do not use it.

On Mar 10, 2017 10:03 AM, "shyla deshpande" <deshpandeshyla@gmail.com>
wrote:

> I have a spark streaming application which processes 3 kafka streams and
> has 5 output operations.
>
> Not sure what should be the setting for spark.streaming.concurrentJobs.
>
> 1. If the concurrentJobs setting is 4 does that mean 2 output operations
> will be run sequentially?
>
> 2. If I had 6 cores what would be a ideal setting for concurrentJobs in
> this situation?
>
> I appreciate your input. Thanks
>

Mime
View raw message