spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: println in spark-shell
Date Sun, 17 May 2015 10:07:28 GMT
println does not execute in the driver since you are executing it on
elements of the RDD. It executes in an executor, which can happen to
execute in-process in local mode. In general you should not expect
this to print results in the driver.

On Sun, May 17, 2015 at 10:01 AM, xiaohe lan <zombiexcoder@gmail.com> wrote:
> Hi,
>
> When I start spark shell by passing yarn to master option, println does not
> print elements in RDD:
>
> bash-4.1$ spark-shell --master yarn
> 15/05/17 01:50:08 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Welcome to
>       ____              __
>      / __/__  ___ _____/ /__
>     _\ \/ _ \/ _ `/ __/  '_/
>    /___/ .__/\_,_/_/ /_/\_\   version 1.3.1
>       /_/
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_45)
> Type in expressions to have them evaluated.
> Type :help for more information.
> Spark context available as sc.
> SQL context available as sqlContext.
>
> scala> val lines = sc.parallelize(List("hello world", "hi"))
> lines: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at
> parallelize at <console>:21
>
> scala> lines.first()
> res1: String = hello world
>
> scala> lines.foreach(println)
>
> scala>
>
> If I start spark shell in local mode, the elements are printed. What's the
> difference here ?
>
> Thanks,
> Xiaohe

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message