spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: httpclient conflict in spark
Date Thu, 30 Mar 2017 13:55:21 GMT

On 29 Mar 2017, at 14:42, Arvind Kandaswamy <aravind.kanda@gmail.com<mailto:aravind.kanda@gmail.com>>
wrote:

Hello,

I am getting the following error. I get this error when trying to use AWS S3. This appears
to be a conflict with httpclient. AWS S3 comes with httplient-4.5.2.jar. I am not sure how
to force spark to use this version. I have tried spark.driver.userClassPathFirst = true, spark.executor.userClassPathFirst=true.
Did not help. I am using Zeppelin to call the spark engine in case if that is an issue.


Spark 2.x ships with httpclient 4.5.2; there is no conflict there. if you are on 1.6, you
could actually try bumping the httplient and httpcomponents  to be consistent

    <!--  org.apache.httpcomponents/httpclient-->
    <commons.httpclient.version>4.5.2</commons.httpclient.version>
    <commons.httpcore.version>4.4.4</commons.httpcore.version>

its important to have org.apache.httpcomponents / httpcore in sync with httpclient; it's probably
there where your problem is arising

Is there anything else that I can try?

java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:51)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:39)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:314)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:298)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:165)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:583)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:563)
at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:541)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:235)


Mime
View raw message