spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Chan <>
Subject Spark Streaming - How to control the parallelism like storm
Date Tue, 22 Oct 2013 14:24:47 GMT
In storm, you can control the number of thread with the setSpout/setBolt,
and how to do the same with Spark Streaming?


val lines = ssc.socketTextStream(args(1), args(2).toInt)
val words = lines.flatMap(_.split(" "))
val wordCounts = => (x, 1)).reduceByKey(_ + _)

Sound like I cannot tell Spark to tell how many thread to be used with
`flatMap` and how many thread to be used with `map` etc, right?

View raw message