spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: UnusedStubClass in 1.3.0-rc1
Date Wed, 25 Feb 2015 18:28:05 GMT
I'm building with

mvn -Phadoop-2.4 -DskipTests install


Yeah, commenting out the unused dependency in the root pom.xml resolves
it.  I'm a little surprised that it cropped up now as well, I had built
against multiple different snapshots of 1.3 over the past couple weeks with
no problems.


Is it worth me putting in a PR to remove that class and the dependency
altogether?

On Wed, Feb 25, 2015 at 11:42 AM, Patrick Wendell <pwendell@gmail.com>
wrote:

> This has been around for multiple versions of Spark, so I am a bit
> surprised to see it not working in your build.
>
> - Patrick
>
> On Wed, Feb 25, 2015 at 9:41 AM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> > Hey Cody,
> >
> > What build command are you using? In any case, we can actually comment
> > out the "unused" thing now in the root pom.xml. It existed just to
> > ensure that at least one dependency was listed in the shade plugin
> > configuration (otherwise, some work we do that requires the shade
> > plugin does not happen). However, now there are other things there. If
> > you just comment out the line in the root pom.xml adding this
> > dependency, does it work?
> >
> > - Patrick
> >
> > On Wed, Feb 25, 2015 at 7:53 AM, Cody Koeninger <cody@koeninger.org>
> wrote:
> >> So when building 1.3.0-rc1 I see the following warning:
> >>
> >> [WARNING] spark-streaming-kafka_2.10-1.3.0.jar, unused-1.0.0.jar define
> 1
> >> overlappping classes:
> >>
> >> [WARNING]   - org.apache.spark.unused.UnusedStubClass
> >>
> >>
> >> and when trying to build an assembly of a project that was previously
> using
> >> 1.3 snapshots without difficulty, I see the following errors:
> >>
> >>
> >> [error] (*:assembly) deduplicate: different file contents found in the
> >> following:
> >>
> >> [error]
> >>
> /Users/cody/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.3.0/spark-streaming-kafka_2.10-1.3.0.jar:org/apache/spark/unused/UnusedStubClass.class
> >>
> >> [error]
> >>
> /Users/cody/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
> >>
> >>
> >> This persists even after a clean / rebuild of both 1.3.0-rc1 and the
> >> project using it.
> >>
> >>
> >> I can just exclude that jar in the assembly definition, but is anyone
> else
> >> seeing similar issues?  If so, might be worth resolving rather than make
> >> users mess with assembly exclusions.
> >>
> >> I see that this class was introduced a while ago, related to SPARK-3812
> but
> >> the jira issue doesn't have much detail.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message