spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simone Pallotta <>
Subject different behaviour linux/Unix vs windows when load spark context in scala method called from R function using rscala package
Date Sun, 27 Aug 2017 07:46:53 GMT
In my R code, I am using rscala package to bridge to a scala method. in scala method I have
initialized a spark context to be used later.

R code:

s <- scala(classpath = "<path to jar>", heap.maximum = "4g")



where init is a scala function and Wrapper is scala class name (see below).

I created a scala object (Wrapper) with init function

scala code:

package it.wrapper.r

object Wrapper {

  var Spark_context: SparkContext = _

    def init(): Unit = {

        val spark_conf = new SparkConf()



          .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

          .set("spark.executor.memory", "6g")

          .set("spark.driver.memory", "2g")

        Spark_context = SparkContext.getOrCreate(spark_conf)



I removed the following code after Spark_context because is not useful.

this code works without flaws under OS X and Linux. Of course window is not gentle as other
OSes. I checked out my project under windows (I have installed scala 2.11.8, java 1.8 and
create environment variable JAVA_HOME and SCALA_HOME that are mandatory for running rscala
package in R, I did the same in OS X/Linux)

when I run the jar file as standalone application it works fine under windows, but when I
invoke the same function from R it fails giving this error:

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class

        at org.apache.hadoop.conf.Configuration.getClass(






        at `enter code here`

        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)

        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:295)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:145)

        at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98)

        at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(<console>:19)

        at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16)

        at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16)

        ... 46 more

    Caused by: java.lang.RuntimeException: class

        at org.apache.hadoop.conf.Configuration.getClass(

        ... 62 more


    java.lang.RuntimeException: java.lang.RuntimeException: class

Does anyone has already seen this kind of behavior?

View raw message