spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhijeet Kumar <abhijeet.ku...@sentienz.com>
Subject Spark streaming join on yarn
Date Wed, 28 Nov 2018 22:25:47 GMT
Hello Team,

I’m doing a simple join. I’m running Spark on Yarn and performing a simple two streaming
join.


DAG of my job




So, it’s taking around 13 secs to complete stage 2.

My command to run jar:

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.1.0 --class com.streaming.spark.RealtimeTuning
--master yarn --deploy-mode cluster --executor-memory 4G --driver-memory 2G --num-executors
1 ./target/scala-2.11/RealtimeTuning-assembly-0.1.jar

Note: I’m running everything locally(Single node cluster)

Any help would be appreciated.

Thank you,

Abhijeet Kumar
Software Development Engineer
Sentienz Solutions Private Limited
Mime
View raw message