spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff - Data Bean Australia <databean...@gmail.com>
Subject Re: Please help with external package using --packages option in spark-shell
Date Tue, 09 Feb 2016 07:21:07 GMT
Finally I figured out the problem and fixed it.

There was some inconsistency in my .ivy2 and .m2 repository. Spark resolves
the dependencies using meta data in ivy2/cache by not verifies its real
location. That was why Spark resolved jackson-core-asl in local-m2-cache.
But when Spark tried to load the JAR file, it couldn't find it and caused
the error.

To solve the problem, I deleted the corresponding directory for
jackson-core-asl under .ivy2.


On Fri, Feb 5, 2016 at 5:00 PM, Jeff - Data Bean Australia <
databean.au@gmail.com> wrote:

> Hi,
>
> Trying to use an external package with the following command line and it
> doesn't work:
>
> ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10
>
> Please find the package info here:
> http://spark-packages.org/package/harsha2010/magellan
>
> The error related to one of the dependencies was not found successfully:
>
> [NOT FOUND  ]
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>
> And the weird part is that Spark or Ivy didn't actually try to look it up
> online, but simply reported error when it didn't find the package in my
> local m2 repository. Please see the complete error message below and help
> me out.
>
> MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
> harsha2010:magellan:1.0.3-s_2.10
> Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
> The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
> :: loading settings :: url =
> jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> harsha2010#magellan added as a dependency
> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
> confs: [default]
> found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
> found commons-io#commons-io;2.4 in local-m2-cache
> found com.esri.geometry#esri-geometry-api;1.2.1 in central
> found org.json#json;20090211 in local-m2-cache
> found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
> :: resolution report :: resolve 282ms :: artifacts dl 9ms
> :: modules in use:
> com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
> commons-io#commons-io;2.4 from local-m2-cache in [default]
> harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
> org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
> [default]
> org.json#json;20090211 from local-m2-cache in [default]
> ---------------------------------------------------------------------
> |                  |            modules            ||   artifacts   |
> |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
> ---------------------------------------------------------------------
> |      default     |   5   |   0   |   0   |   0   ||   5   |   0   |
> ---------------------------------------------------------------------
>
> :: problems summary ::
> :::: WARNINGS
> [NOT FOUND  ]
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>
> ==== local-m2-cache: tried
>
>
>  file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
> ::              FAILED DOWNLOADS            ::
>
> :: ^ see resolution messages for details  ^ ::
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
> :: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar
>
> ::::::::::::::::::::::::::::::::::::::::::::::
>
>
>
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> Exception in thread "main" java.lang.RuntimeException: [download failed:
> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
> at
> org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> --
> Data Bean - A Big Data Solution Provider in Australia.
>



-- 
Data Bean - A Big Data Solution Provider in Australia.

Mime
View raw message