spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Chan <ryanchan...@gmail.com>
Subject Spark Streaming - How to control the parallelism like storm
Date Tue, 22 Oct 2013 14:24:47 GMT
In storm, you can control the number of thread with the setSpout/setBolt,
and how to do the same with Spark Streaming?

e.g.

val lines = ssc.socketTextStream(args(1), args(2).toInt)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()


Sound like I cannot tell Spark to tell how many thread to be used with
`flatMap` and how many thread to be used with `map` etc, right?

Mime
View raw message