spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: spark-submit command-line with --files
Date Fri, 19 Sep 2014 18:01:02 GMT
Hi Chinchu,

SparkEnv is an internal class that is only meant to be used within Spark.
Outside of Spark, it will be null because there are no executors or driver
to start an environment for. Similarly, SparkFiles is meant to be used
internally (though it's privacy settings should be modified to reflect
that). Is there a reason why you need to pass the serialized objects this
way? Can't you access the deserialized form from your application?

Andrew

2014-09-18 22:53 GMT-07:00 chinchu <chinchu.sup@gmail.com>:

> Hi,
>
> I am running spark-1.1.0 and I want to pass in a file (that contains java
> serialized objects used to initialize my program) to the App main program.
> I
> am using the --files option but I am not able to retrieve the file in the
> main_class. It reports a null pointer exception. [I tried both local &
> yarn-cluster with the same result]. I am using the
> SparkFiles.get("myobject.ser") to get the file. Am I doing something wrong
> ?
>
> CMD:
> bin/spark-submit --name  Test --class
> com.test.batch.modeltrainer.ModelTrainerMain \
>   --master local --files /tmp/myobject.ser --verbose
> /opt/test/lib/spark-test.jar
>
> com.test.batch.modeltrainer.ModelTrainerMain.scala
> 37: val serFile = SparkFiles.get("myobject.ser")
>
> Exception:
> Exception in thread "main" java.lang.NullPointerException
>   at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37)
>   at org.apache.spark.SparkFiles$.get(SparkFiles.scala:31)
>   at
>
> com.test.batch.modeltrainer.ModelTrainerMain$.main(ModelTrainerMain.scala:37)
>   at
> com.test.batch.modeltrainer.ModelTrainerMain.main(ModelTrainerMain.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Looking at the Scala code for SparkFiles:37, it looks like SparkEnv.get is
> getting a null ..
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-command-line-with-files-tp14645.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message