spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinodkc <vinod.kc...@gmail.com>
Subject spark shell paste mode is not consistent
Date Thu, 16 Apr 2015 10:36:38 GMT
Hi All,

I faced below issue while working with spark. It seems spark shell paste
mode is not consistent

Example code 
---------------
val textFile = sc.textFile("README.md")
textFile.count() 
textFile.first() 
val linesWithSpark = textFile.filter(line => line.contains("Spark"))
textFile.filter(line => line.contains("Spark")).count() 

Step 1 : Run above code in spark-shell
--------

scala> val textFile = sc.textFile("README.md")
textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1]
at textFile at <console>:21

scala> textFile.count() 
res0: Long = 98

scala> textFile.first() 
res1: String = # Apache Spark

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
linesWithSpark: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at
filter at <console>:23

scala> textFile.filter(line => line.contains("Spark")).count() 
res2: Long = 19

Result 1: Following actions are evaluated properly
textFile.count() ,textFile.first() ,textFile.filter(line =>
line.contains("Spark")).count() 
res0: Long = 98,res1: String = # Apache Spark,res2: Long = 19

Step 2 : Run above code in spark-shell paste mode
scala> :p
// Entering paste mode (ctrl-D to finish)

val textFile = sc.textFile("README.md")
textFile.count() 
textFile.first() 
val linesWithSpark = textFile.filter(line => line.contains("Spark"))
textFile.filter(line => line.contains("Spark")).count()

// Exiting paste mode, now interpreting.

textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1]
at textFile at <console>:21
linesWithSpark: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at
filter at <console>:24
res0: Long = 19

scala>

Result 2: Only one action is executed 
textFile.filter(line => line.contains("Spark")).count()  
res0: Long = 19

Expected result : Result 1 and Result 2 should be same

I feel this is an issue with spark shell . I fixed and verified it
locally.If community also think that it need to be handled, I can make a PR.

Thanks
Vinod KC



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-paste-mode-is-not-consistent-tp11621.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message