spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hrishikesh Mishra <sd.hri...@gmail.com>
Subject What is the best way to consume parallely from multiple topics in Spark Stream with Kafka
Date Wed, 04 Mar 2020 18:42:14 GMT
Hi

My spark stream job consumes from multiple Kafka topics. How can I process
parallely? Should I try for *spark.streaming.concurrentJobs,* but it has
some adverse effects as mentioned by the creator. Is it still valid with
Spark 2.4 and Direct Kafka Stream? What about FAIR scheduling mode, will it
help in this scenario. I am not getting any valid links around this.

Regards
Hrishi

Mime
View raw message