spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1
Date Tue, 25 Mar 2014 23:10:22 GMT
Starting with Spark 0.9 the protobuf dependency we use is shaded and
cannot interfere with other protobuf libaries including those in
Hadoop. Not sure what's going on in this case. Would someone who is
having this problem post exactly how they are building spark?

- Patrick

On Fri, Mar 21, 2014 at 3:49 PM, Aureliano Buendia <buendia360@gmail.com> wrote:
>
>
>
> On Tue, Mar 18, 2014 at 12:56 PM, Ognen Duzlevski
> <ognen@plainvanillagames.com> wrote:
>>
>>
>> On 3/18/14, 4:49 AM, dmpour23@gmail.com wrote:
>>>
>>> On Sunday, 2 March 2014 19:19:49 UTC+2, Aureliano Buendia  wrote:
>>>>
>>>> Is there a reason for spark using the older akka?
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Mar 2, 2014 at 1:53 PM, 1esha <alexey.r...@gmail.com> wrote:
>>>>
>>>> The problem is in akka remote. It contains files compiled with 2.4.*.
>>>> When
>>>>
>>>> you run it with 2.5.* in classpath it fails like above.
>>>>
>>>>
>>>>
>>>> Looks like moving to akka 2.3 will solve this issue. Check this issue -
>>>>
>>>>
>>>> https://www.assembla.com/spaces/akka/tickets/3154-use-protobuf-version-2-5-0#/activity/ticket:
>>>>
>>>>
>>>> Is the solution to exclude the  2.4.*. dependency on protobuf or will
>>>> thi produce more complications?
>>
>> I am not sure I remember what the context was around this but I run 0.9.0
>> with hadoop 2.2.0 just fine.
>
>
> The problem is that spark depends on an older version of akka, which depends
> on an older version of protobuf (2.4).
>
> This means people cannot use protobuf 2.5 with spark.
>
>>
>> Ognen
>
>

Mime
View raw message