spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: [SPARK-3638 ] java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.
Date Mon, 16 Mar 2015 17:06:09 GMT
>From my local maven repo:

$ jar tvf
~/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
| grep SchemeRegistry
  1373 Fri Apr 19 18:19:36 PDT 2013
org/apache/http/impl/conn/SchemeRegistryFactory.class
  2954 Fri Apr 19 18:19:36 PDT 2013
org/apache/http/conn/scheme/SchemeRegistry.class
  2936 Fri Apr 19 18:19:36 PDT 2013
org/apache/http/auth/AuthSchemeRegistry.class

If you run mvn dependency:tree, you would see something similar to the
following:

[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.6.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-common:jar:2.6.0:compile
[INFO] |  |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] |  |  |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |  |  +-
commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  |  \-
commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  |  \-
commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] |  |  |  +- org.apache.avro:avro:jar:1.7.6:compile
[INFO] |  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0:compile
[INFO] |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile

Cheers

On Mon, Mar 16, 2015 at 9:38 AM, Shuai Zheng <szheng.code@gmail.com> wrote:

> Hi All,
>
>
>
> I am running Spark 1.2.1 and AWS SDK. To make sure AWS compatible on the
> httpclient 4.2 (which I assume spark use?), I have already downgrade to the
> version 1.9.0
>
>
>
> But even that, I still got an error:
>
>
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
>
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
>
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
>
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
>
>         at
> com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
>
>         at
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:102)
>
>         at
> com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:190)
>
>         at
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
>
>         at
> com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:410)
>
>         at
> com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:392)
>
>         at
> com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:376)
>
>
>
> When I search the maillist, it looks the same issue as:
>
> https://github.com/apache/spark/pull/2535
>
>
> http://stackoverflow.com/questions/24788949/nosuchmethoderror-while-running-aws-s3-client-on-spark-while-javap-shows-otherwi
>
>
>
> But I don’t understand the solution mention here? The issue is caused by
> an pre-package  DefaultClientConnectionOperator in the spark all-in-one
> jar file which doesn’t have the that method.
>
>
>
> I have some questions here:
>
>
>
> How can we find out which exact version when spark try to pre-package
> everything (this really very painful). and how can we override it?
>
>
>
> I have tried:
>
>
>
>     *val* conf = *new* SparkConf()
>
>       .set("spark.files.userClassPathFirst", "true")// For non Yarn APP
> before spark 1.3
>
>       .set("spark.executor.userClassPathFirst", "true")// For spark 1.3.0
>
> But it doesn’t work
>
>
>
> This really create a lot of issues to me (especially we don’t know what
> version is used by Spark to package its own jar, we need to try out). Even
> maven doesn’t give enough information because httpclient is not under the
> maven dependency (even indirect dependency, after I use tools to resolved
> the whole dependency tree).
>
>
>
> Regards,
>
>
>
> Shuai
>

Mime
View raw message