spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: Serializable incompatible with Externalizable error
Date Tue, 03 Dec 2013 19:31:51 GMT
This discussion seems to indicate the possibility of a mismatch between one
side being Serializable and the other being Externalizable:
https://forums.oracle.com/thread/2147644

In general, the semantics of Serializable can be pretty strange as it
doesn't really behave the same as usual interfaces. It may actually be
possible that if a superclass implements Serializable and you implement
Externalizable, things can screw up, for instance.


On Tue, Dec 3, 2013 at 10:48 AM, Matt Cheah <mcheah@palantir.com> wrote:

>  That can't be – Externalizable is a specific sub-interface of
> Serializable that allows custom serialization formats.
>
>  Granted, I could accomplish the same thing by implementing the private
> readObject() and writeObject() methods in this class, and implement
> Serializable only. It just seems odd to me that I'd have to do so.
> Especially since the tuning guide suggests to use Externalizable:
> http://spark.incubator.apache.org/docs/latest/tuning.html
>
>  -Matt Cheah
>
>   From: Andrew Ash <andrew@andrewash.com>
> Reply-To: "user@spark.incubator.apache.org" <
> user@spark.incubator.apache.org>
> Date: Monday, December 2, 2013 10:45 PM
> To: "user@spark.incubator.apache.org" <user@spark.incubator.apache.org>
> Subject: Re: Serializable incompatible with Externalizable error
>
>   At least from
> http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java<https://urldefense.proofpoint.com/v1/url?u=http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=gxvgJndY02bAG2cHbPl1cUTcd%2FLzFGz7wtfiAfRKPpk%3D%0A&m=1AzN%2FqpvE4RUjLF0SQcqdBp8GpAKxHbhF7JFlnvwkHI%3D%0A&s=ca3c6dbf7a4045601fbd4b5634863075aca6f5dda2606bb01b34eb47f49eb225>it
looks like Externalizable is roughly an old-java version of
> Serializable.  Does that class implement both interfaces?  Can you take
> away the Externalizable interface if it's being used?
>
>
> On Mon, Dec 2, 2013 at 7:15 PM, Matt Cheah <mcheah@palantir.com> wrote:
>
>>  Hi everyone,
>>
>>  I'm running into a case where I'm creating a Java RDD of an
>> Externalizable class, and getting this stack trace:
>>
>>  java.io.InvalidClassException (java.io.InvalidClassException:
>> com.palantir.finance.datatable.server.spark.WritableDataRow; Serializable
>> incompatible with Externalizable)
>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634)
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>> java.io.ObjectInputStream.readClass(ObjectInputStream.java:1483)<some
>> other Java stuff>
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
>> I'm running on a spark cluster generated by the EC2 Scripts. This doesn't
>> happen if I'm running things with local[N]. Any ideas?
>> Thanks,
>> -Matt Cheah
>>
>
>

Mime
View raw message