spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandeep Singh ...@chandeep.com>
Subject Re: Error building a self contained Spark app
Date Fri, 04 Mar 2016 23:25:53 GMT
This is what you need:

    val sc = new SparkContext(sparkConf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.implicits._

> On Mar 4, 2016, at 11:03 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
> 
> Hi Ted,
> 
> This is my code
> 
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.SQLContext
> //
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new SparkConf().setAppName("Sequence").setMaster("local[*]").set("spark.driver.allowMultipleContexts",
"true")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> And the error I am getting now is
> 
> [info] downloading https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar
<https://repo1.maven.org/maven2/org/scala-lang/jline/2.10.5/jline-2.10.5.jar> ...
> [info]  [SUCCESSFUL ] org.scala-lang#jline;2.10.5!jline.jar (103ms)
> [info] Done updating.
> [info] Compiling 1 Scala source to /home/hduser/dba/bin/scala/Sequence/target/scala-2.10/classes...
> [info] 'compiler-interface' not yet compiled for Scala 2.10.5. Compiling...
> [info]   Compilation completed in 12.462 s
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: value toDF
is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error]             ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:17: not found:
value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:19: value toDF
is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error]     ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 88 s, completed Mar 4, 2016 11:12:46 PM
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>  
> 
> On 4 March 2016 at 22:52, Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com>>
wrote:
> Can you show your code snippet ?
> Here is an example:
> 
>       val sqlContext = new SQLContext(sc)
>       import sqlContext.implicits._
> 
> On Fri, Mar 4, 2016 at 1:55 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> Hi Ted,
> 
>  I am getting the following error after adding that import
> 
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:5: not found:
object sqlContext
> [error] import sqlContext.implicits._
> [error]        ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value toDF
is not a member of Seq[(String, Int)]
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>  
> 
> On 4 March 2016 at 21:39, Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com>>
wrote:
> Can you add the following into your code ?
>  import sqlContext.implicits._
> 
> On Fri, Mar 4, 2016 at 1:14 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> Hi,
> 
> I have a simple Scala program as below
> 
> import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.SQLContext
> object Sequence {
>   def main(args: Array[String]) {
>   val conf = new SparkConf().setAppName("Sequence")
>   val sc = new SparkContext(conf)
>   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>   val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>   val a = Seq(("Mich",20), ("Christian", 18), ("James",13), ("Richard",16))
>   // Sort option 1 using tempTable
>   val b = a.toDF("Name","score").registerTempTable("tmp")
>   sql("select Name,score from tmp order by score desc").show
>   // Sort option 2 with FP
>   a.toDF("Name","score").sort(desc("score")).show
>  }
> }
> 
> I build this using sbt tool as below
> 
>  cat sequence.sbt
> name := "Sequence"
> version := "1.0"
> scalaVersion := "2.10.5"
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.0"
> 
> 
> But it fails compilation as below
> 
> [info]   Compilation completed in 12.366 s
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:15: value toDF
is not a member of Seq[(String, Int)]
> [error]   val b = a.toDF("Name","score").registerTempTable("tmp")
> [error]             ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:16: not found:
value sql
> [error]   sql("select Name,score from tmp order by score desc").show
> [error]   ^
> [error] /home/hduser/dba/bin/scala/Sequence/src/main/scala/Sequence.scala:18: value toDF
is not a member of Seq[(String, Int)]
> [error]   a.toDF("Name","score").sort(desc("score")).show
> [error]     ^
> [error] three errors found
> [error] (compile:compileIncremental) Compilation failed
> [error] Total time: 95 s, completed Mar 4, 2016 9:06:40 PM
> 
> I think I am missing some dependencies here
> 
> 
> I have a simple 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>  
> 
> 
> 
> 


Mime
View raw message