spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Malouf <malouf.g...@gmail.com>
Subject Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1
Date Tue, 25 Mar 2014 21:47:46 GMT
Can anyone verify the claims from Aureliano regarding the Akka dependency
protobuf collision?  Our team has a major need to upgrade to protobuf 2.5.0
up the pipe and Spark seems to be the blocker here.


On Fri, Mar 21, 2014 at 6:49 PM, Aureliano Buendia <buendia360@gmail.com>wrote:

>
>
>
> On Tue, Mar 18, 2014 at 12:56 PM, Ognen Duzlevski <
> ognen@plainvanillagames.com> wrote:
>
>>
>> On 3/18/14, 4:49 AM, dmpour23@gmail.com wrote:
>>
>>> On Sunday, 2 March 2014 19:19:49 UTC+2, Aureliano Buendia  wrote:
>>>
>>>> Is there a reason for spark using the older akka?
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Mar 2, 2014 at 1:53 PM, 1esha <alexey.r...@gmail.com> wrote:
>>>>
>>>> The problem is in akka remote. It contains files compiled with 2.4.*.
>>>> When
>>>>
>>>> you run it with 2.5.* in classpath it fails like above.
>>>>
>>>>
>>>>
>>>> Looks like moving to akka 2.3 will solve this issue. Check this issue -
>>>>
>>>> https://www.assembla.com/spaces/akka/tickets/3154-use-
>>>> protobuf-version-2-5-0#/activity/ticket:
>>>>
>>>>
>>>> Is the solution to exclude the  2.4.*. dependency on protobuf or will
>>>> thi produce more complications?
>>>>
>>> I am not sure I remember what the context was around this but I run
>> 0.9.0 with hadoop 2.2.0 just fine.
>>
>
> The problem is that spark depends on an older version of akka, which
> depends on an older version of protobuf (2.4).
>
> This means people cannot use protobuf 2.5 with spark.
>
>
>> Ognen
>>
>
>

Mime
View raw message