spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kelly, Jonathan" <jonat...@amazon.com>
Subject Spark v1.2.1 failing under BigTop build in External Flume Sink (due to missing Netty library)
Date Thu, 05 Mar 2015 18:39:38 GMT
I'm running into an issue building Spark v1.2.1 (as well as the latest in branch-1.2 and v1.3.0-rc2
and the latest in branch-1.3) with BigTop (v0.9, which is not quite released yet).  The build
fails in the External Flume Sink subproject with the following error:

[INFO] Compiling 5 Scala sources and 3 Java sources to /workspace/workspace/bigtop.spark-rpm/build/spark/rpm/BUILD/spark-1.3.0/external/flume-sink/target/scala-2.10/classes...
[WARNING] Class org.jboss.netty.channel.ChannelFactory not found - continuing with a stub.
[ERROR] error while loading NettyServer, class file '/home/ec2-user/.m2/repository/org/apache/avro/avro-ipc/1.7.6/avro-ipc-1.7.6.jar(org/apache/avro/ipc/NettyServer.class)'
is broken
(class java.lang.NullPointerException/null)
[WARNING] one warning found
[ERROR] one error found

It seems like what is happening is that the Netty library is missing at build time, which
happens because it is explicitly excluded in the pom.xml (see https://github.com/apache/spark/blob/v1.2.1/external/flume-sink/pom.xml#L42).
 I attempted removing the exclusions and the explicit re-add for the test scope on lines 77-88,
and that allowed the build to succeed, though I don't know if that will cause problems at
runtime.  I don't have any experience with the Flume Sink, so I don't really know how to test
it.  (And, to be clear, I'm not necessarily trying to get the Flume Sink to work-- I just
want the project to build successfully, though of course I'd still want the Flume Sink to
work for whomever does need it.)

Does anybody have any idea what's going on here?  Here is the command BigTop is running to
build Spark:

mvn -Pbigtop-dist -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl -Divy.home=/home/ec2-user/.ivy2
-Dsbt.ivy.home=/home/ec2-user/.ivy2 -Duser.home=/home/ec2-user -Drepo.maven.org= -Dreactor.repo=file:///home/ec2-user/.m2/repository
-Dhadoop.version=2.4.0-amzn-3-SNAPSHOT -Dyarn.version=2.4.0-amzn-3-SNAPSHOT -Dprotobuf.version=2.5.0
-Dscala.version=2.10.3 -Dscala.binary.version=2.10 -DskipTests -DrecompileMode=all install

As I mentioned above, if I switch to the latest in branch-1.2, to v1.3.0-rc2, or to the latest
in branch-1.3, I get the same exact error.  I was not getting the error with Spark v1.1.0,
though there weren't any changes to the external/flume-sink/pom.xml between v1.1.0 and v1.2.1.

~ Jonathan Kelly

Mime
View raw message