spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu, Raymond" <>
Subject issue regarding akka, protobuf and Hadoop version
Date Mon, 04 Nov 2013 07:46:08 GMT

	I am working on porting spark onto Hadoop 2.2.0, With some renaming and call into new YARN
API works done. I can run up the spark master. While I encounter the issue that Executor Actor
could not connecting to Driver actor.

	After some investigation, I found the root cause is that the akka-remote do not support protobuf
2.5.0 before 2.3. And hadoop move to protobuf 2.5.0 from 2.1-beta.

	The issue is that if I exclude the akka dependency from hadoop and force protobuf dependency
to 2.4.1, the compile/packing will fail since hadoop common jar require a new interface from
protobuf 2.5.0.

	 So any suggestion on this?

Best Regards,
Raymond Liu

View raw message