beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <>
Subject [jira] [Commented] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts
Date Wed, 01 Feb 2017 12:30:51 GMT


Aviem Zur commented on BEAM-1092:

[~davor] I agree that a general solution requires some thought.

Regarding KafkaIO this should probably be shaded in its pom for now as this is commonly used
in clusters which may have a different version of guava.
I myself had my application crash in a Spark cluster due to KafkaIO not having its guava dependency

While it is true that users can shade their applications themselves it might be too much to
expect them to go through this process, figure out that they have a transient dependency on
a different version of guava then shade it.

> Shade commonly used libraries (e.g. Guava) to avoid class conflicts
> -------------------------------------------------------------------
>                 Key: BEAM-1092
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: examples-java, sdk-java-extensions
>    Affects Versions: 0.3.0-incubating
>            Reporter: Maximilian Michels
>            Assignee: Frances Perry
> Beam shades away some of its dependencies like Guava to avoid user classes from clashing
with these dependencies. Some of the artifacts, e.g. KafkaIO, do not shade any classes and
directly depend on potentially conflicting libraries (e.g. Guava). Also, users might manually
add such libraries as dependencies.
> Runners who add classes to the classpath (e.g. Hadoop) can run into conflict with multiple
versions of the same class. To prevent that, we should adjust the Maven archetypes pom files
used for the Quickstart to perform shading of commonly used libraries (again, Guava is often
the culprit).
> To prevent the problem in the first place, we should expand the shading of Guava and
other libraries to all modules which make use of these. 
> To solve both dimensions of the issue, we need to address:
> 1. Adding shading of commonly used libraries to the archetypes poms
> 2. Properly shade all commonly used libraries in the SDK modules
> 2) seems to be of highest priority since it affects users who simply use the provided
IO modules.

This message was sent by Atlassian JIRA

View raw message