spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff - Data Bean Australia <databean...@gmail.com>
Subject Please help with external package using --packages option in spark-shell
Date Fri, 05 Feb 2016 04:00:21 GMT
Hi,

Trying to use an external package with the following command line and it
doesn't work:

./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10

Please find the package info here:
http://spark-packages.org/package/harsha2010/magellan

The error related to one of the dependencies was not found successfully:

[NOT FOUND  ]
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)

And the weird part is that Spark or Ivy didn't actually try to look it up
online, but simply reported error when it didn't find the package in my
local m2 repository. Please see the complete error message below and help
me out.

MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
harsha2010:magellan:1.0.3-s_2.10
Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
:: loading settings :: url =
jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
harsha2010#magellan added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
found commons-io#commons-io;2.4 in local-m2-cache
found com.esri.geometry#esri-geometry-api;1.2.1 in central
found org.json#json;20090211 in local-m2-cache
found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
:: resolution report :: resolve 282ms :: artifacts dl 9ms
:: modules in use:
com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
commons-io#commons-io;2.4 from local-m2-cache in [default]
harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
[default]
org.json#json;20090211 from local-m2-cache in [default]
---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |   5   |   0   |   0   |   0   ||   5   |   0   |
---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
[NOT FOUND  ]
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)

==== local-m2-cache: tried

 file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar

::::::::::::::::::::::::::::::::::::::::::::::

::              FAILED DOWNLOADS            ::

:: ^ see resolution messages for details  ^ ::

::::::::::::::::::::::::::::::::::::::::::::::

:: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar

::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed:
org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

-- 
Data Bean - A Big Data Solution Provider in Australia.

Mime
View raw message