spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: How to add elements into map?
Date Mon, 10 Nov 2014 00:56:37 GMT
Hi,

On Sat, Nov 8, 2014 at 1:39 PM, Tim Chou <timchou.hit@gmail.com> wrote:
>
> val table = sc.textFile(args(1))
> val histMap = collection.mutable.Map[Int,Int]()
> for (x <- table) {
>
>
>         val tuple = x.split('|')
>
>
>         histMap.put(tuple(0).toInt, 1)
>
>
>     }
>

What will happen here is that histMap (an empty Map) will be serialized and
sent to all Spark workers. Each worker will fill it locally with the data
that was processed locally, but it won't ever be sent back to the Spark
driver, that's why you don't see anything there.

Tobias

Mime
View raw message