spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: regression: no longer able to use HDFS wasbs:// path for additional python files on LIVY batch submit
Date Mon, 03 Oct 2016 12:40:24 GMT

On 1 Oct 2016, at 02:49, Kevin Grealish <<>>

I’m seeing a regression when submitting a batch PySpark program with additional files using
LIVY. This is YARN cluster mode. The program files are placed into the mounted Azure Storage
before making the call to LIVY. This is happening from an application which has credentials
for the storage and the LIVY endpoint, but not local file systems on the cluster. This previously
worked but now I’m getting the error below.

Seems this restriction was introduced with
(new in 1.6.2 and 2.0.0).

How should the scenario above be achieved now? Am I missing something?

This has been fixed in ; I don't know if
its in 2.0.1 though

Exception in thread "main" java.lang.IllegalArgumentException: Launching Python applications
through spark-submit is currently only supported for local files: wasb://
                at org.apache.spark.deploy.PythonRunner$.formatPath(PythonRunner.scala:104)
                at org.apache.spark.deploy.PythonRunner$$anonfun$formatPaths$3.apply(PythonRunner.scala:136)
                at org.apache.spark.deploy.PythonRunner$$anonfun$formatPaths$3.apply(PythonRunner.scala:136)
                at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
                at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
                at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
                at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
                at scala.collection.TraversableLike$
                at scala.collection.mutable.ArrayOps$
                at org.apache.spark.deploy.PythonRunner$.formatPaths(PythonRunner.scala:136)
                at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$11.apply(SparkSubmit.scala:639)
                at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$11.apply(SparkSubmit.scala:637)
                at scala.Option.foreach(Option.scala:236)
                at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:637)
                at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
                at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
                at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.Exception: spark-submit exited with code 1}.

View raw message