spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 浦野 裕也 <y-ur...@kitadenshi.co.jp>
Subject Re: AWS SDK HttpClient version conflict (spark.files.userClassPathFirst not working)
Date Fri, 13 Mar 2015 01:22:11 GMT
Hi Adam,

Could you try building spark with profile -Pkinesis-asl.
mvn -Pkinesis-asl -DskipTests clean package

refers to 'Running the Example' section.
https://spark.apache.org/docs/latest/streaming-kinesis-integration.html

In fact, I've seen same issue and have been able to use the AWS SDK by 
trying the above.
I've also tried to use 'spark.files.userClassPathFirst' flag, and it 
doesn't work.

Regards,
Yuya Urano


On 2015/03/13 3:50, Adam Lewandowski wrote:
> I'm trying to use the AWS SDK (v1.9.23) to connect to DynamoDB from 
> within a Spark application. Spark 1.2.1 is assembled with HttpClient 
> 4.2.6, but the AWS SDK is depending on HttpClient 4.3.4 for it's 
> communication with DynamoDB. The end result is an error when the app 
> tries to connect to DynamoDB and gets Spark's version instead:
> java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpPatch
> at com.amazonaws.http.AmazonHttpClient.<clinit>(AmazonHttpClient.java:129)
> at 
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
> at 
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:359)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.http.client.methods.HttpPatch
>
> Including HttpClient 4.3.4 as user jars doesn't improve the situation 
> much:
> java.lang.NoSuchMethodError: 
> org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
> at 
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
>
> I've seen the documenation regarding the 
> 'spark.files.userClassPathFirst' flag and have tried to use it 
> thinking it would resolve this issue. However, when that flag is used 
> I get an NoClassDefFoundError on 'scala.Serializable':
> java.lang.NoClassDefFoundError: scala/Serializable
> ...
> at 
> org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:46)
> ...
> Caused by: java.lang.ClassNotFoundException: scala.Serializable
>
> This seems odd to me, since scala.Serializable is included in the 
> spark assembly. I thought perhaps my app was compiled against a 
> different scala version than spark uses, but eliminated that 
> possibility by using the scala compiler directly out of the spark 
> assembly jar with identical results.
>
> Has anyone else seen this issue, had any success with the 
> "spark.files.userClassPathFirst" flag, or been able to use the AWS SDK?
> I was going to submit this a Spark JIRA issue, but thought I would 
> check here first.
>
> Thanks,
> Adam Lewandowski
>
>

Mime
View raw message