Hey,
Spark Submit adds maven central & spark bintray to the ChainResolver before it adds any
external resolvers. https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821
When running on a cluster without internet access, this means the spark shell takes forever
to launch as it tries these two remote repos before the ones specified in the --repositories
list. In our case we have a proxy which the cluster can access it and supply it via —repositories.
This is also a problem for users who maintain a proxy for maven/ivy repos with something like
Nexus/Artifactory.
I see two options for a fix;
* Change the order repos are added to the ChainResolver, making the --repositories supplied
repos come before anything else. https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843
* Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default true) which
when false wont add the maven central & bintry to the ChainResolver.
Happy to do a PR now for this if someone can give me a recommendation on which option would
be better.
JIRA here; https://issues.apache.org/jira/browse/SPARK-8475
Cheers,
Nathan
|