spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff - Data Bean Australia <databean...@gmail.com>
Subject Re: Please help with external package using --packages option in spark-shell
Date Fri, 05 Feb 2016 04:20:09 GMT
Thanks Divya for helping me.

It does connect to the internet.

And I even tried to use a local artifactory repository and it didn't work
either.

$ ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10 --repositories
http://localhost:8081/artifactory/libs-release
Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
http://localhost:8081/artifactory/libs-release added as a remote repository
with the name: repo-1
:: loading settings :: url =
jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
harsha2010#magellan added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
found commons-io#commons-io;2.4 in local-m2-cache
found com.esri.geometry#esri-geometry-api;1.2.1 in central
found org.json#json;20090211 in local-m2-cache
found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
:: resolution report :: resolve 282ms :: artifacts dl 10ms
:: modules in use:
com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
commons-io#commons-io;2.4 from local-m2-cache in [default]
harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
[default]
org.json#json;20090211 from local-m2-cache in [default]

On Fri, Feb 5, 2016 at 5:01 PM, Divya Gehlot <divya.htconex@gmail.com>
wrote:

> Is your machine connected to internet ?
>
>
> On 5 February 2016 at 12:00, Jeff - Data Bean Australia <
> databean.au@gmail.com> wrote:
>
>> Hi,
>>
>> Trying to use an external package with the following command line and it
>> doesn't work:
>>
>> ./spark-shell --packages harsha2010:magellan:1.0.3-s_2.10
>>
>> Please find the package info here:
>> http://spark-packages.org/package/harsha2010/magellan
>>
>> The error related to one of the dependencies was not found successfully:
>>
>> [NOT FOUND  ]
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>>
>> And the weird part is that Spark or Ivy didn't actually try to look it up
>> online, but simply reported error when it didn't find the package in my
>> local m2 repository. Please see the complete error message below and help
>> me out.
>>
>> MacBook-Pro-2:bin jeffhuang$ ./spark-shell --packages
>> harsha2010:magellan:1.0.3-s_2.10
>> Ivy Default Cache set to: /Users/jeffhuang/.ivy2/cache
>> The jars for the packages stored in: /Users/jeffhuang/.ivy2/jars
>> :: loading settings :: url =
>> jar:file:/Users/jeffhuang/learning/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>> harsha2010#magellan added as a dependency
>> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
>> confs: [default]
>> found harsha2010#magellan;1.0.3-s_2.10 in spark-packages
>> found commons-io#commons-io;2.4 in local-m2-cache
>> found com.esri.geometry#esri-geometry-api;1.2.1 in central
>> found org.json#json;20090211 in local-m2-cache
>> found org.codehaus.jackson#jackson-core-asl;1.9.12 in local-m2-cache
>> :: resolution report :: resolve 282ms :: artifacts dl 9ms
>> :: modules in use:
>> com.esri.geometry#esri-geometry-api;1.2.1 from central in [default]
>> commons-io#commons-io;2.4 from local-m2-cache in [default]
>> harsha2010#magellan;1.0.3-s_2.10 from spark-packages in [default]
>> org.codehaus.jackson#jackson-core-asl;1.9.12 from local-m2-cache in
>> [default]
>> org.json#json;20090211 from local-m2-cache in [default]
>> ---------------------------------------------------------------------
>> |                  |            modules            ||   artifacts   |
>> |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
>> ---------------------------------------------------------------------
>> |      default     |   5   |   0   |   0   |   0   ||   5   |   0   |
>> ---------------------------------------------------------------------
>>
>> :: problems summary ::
>> :::: WARNINGS
>> [NOT FOUND  ]
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar (1ms)
>>
>> ==== local-m2-cache: tried
>>
>>
>>  file:/Users/jeffhuang/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>> ::              FAILED DOWNLOADS            ::
>>
>> :: ^ see resolution messages for details  ^ ::
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>> :: org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::
>>
>>
>>
>> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>> Exception in thread "main" java.lang.RuntimeException: [download failed:
>> org.codehaus.jackson#jackson-core-asl;1.9.12!jackson-core-asl.jar]
>> at
>> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1009)
>> at
>> org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> --
>> Data Bean - A Big Data Solution Provider in Australia.
>>
>
>


-- 
Data Bean - A Big Data Solution Provider in Australia.

Mime
View raw message