spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Untangling dependency issues in spark streaming
Date Mon, 30 Mar 2015 00:08:46 GMT
For Gradle, there are:
https://github.com/musketyr/gradle-fatjar-plugin
https://github.com/johnrengelman/shadow

FYI

On Sun, Mar 29, 2015 at 4:29 PM, jay vyas <jayunit100.apache@gmail.com>
wrote:

> thanks for posting this! Ive ran into similar issues before, and generally
> its a bad idea to swap the libraries out and "pray fot the best", so the
> shade functionality is probably the best feature.
>
> Unfortunately, im not sure how well SBT and Gradle support shading... how
> do folks using next gen build tools solve this problem?
>
>
>
> On Sun, Mar 29, 2015 at 3:10 AM, Neelesh <neeleshs@gmail.com> wrote:
>
>> Hi,
>>   My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
>> spark uses 4.2.6 , and I believe thats what's causing the following error.
>> I've tried setting
>> spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to
>> true in the config, but that does not solve it either. Finally I had to
>> resort to relocating classes using maven shade plugin while building my
>> apps uber jar, using
>>
>> <relocations>
>>     <relocation>
>>         <pattern>org.apache.http</pattern>
>>         <shadedPattern>org.shaded.apache.http</shadedPattern>
>>     </relocation>
>> </relocations>
>>
>>
>> Hope this is useful to others in the same situation. It would be really great to
deal with this the right way (like tomcat or any other servlet container - classloader hierarchy
etc).
>>
>>
>> Caused by: java.lang.NoSuchFieldError: INSTANCE
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>>
>> and ...
>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>>
>>
>
>
> --
> jay vyas
>

Mime
View raw message