spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Cheah <mch...@palantir.com>
Subject Re: Serializable incompatible with Externalizable error
Date Tue, 03 Dec 2013 18:48:09 GMT
That can't be – Externalizable is a specific sub-interface of Serializable that allows custom
serialization formats.

Granted, I could accomplish the same thing by implementing the private readObject() and writeObject()
methods in this class, and implement Serializable only. It just seems odd to me that I'd have
to do so. Especially since the tuning guide suggests to use Externalizable: http://spark.incubator.apache.org/docs/latest/tuning.html

-Matt Cheah

From: Andrew Ash <andrew@andrewash.com<mailto:andrew@andrewash.com>>
Reply-To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>"
<user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>>
Date: Monday, December 2, 2013 10:45 PM
To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>>
Subject: Re: Serializable incompatible with Externalizable error

At least from http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java<https://urldefense.proofpoint.com/v1/url?u=http://stackoverflow.com/questions/817853/what-is-the-difference-between-serializable-and-externalizable-in-java&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=gxvgJndY02bAG2cHbPl1cUTcd%2FLzFGz7wtfiAfRKPpk%3D%0A&m=1AzN%2FqpvE4RUjLF0SQcqdBp8GpAKxHbhF7JFlnvwkHI%3D%0A&s=ca3c6dbf7a4045601fbd4b5634863075aca6f5dda2606bb01b34eb47f49eb225>
it looks like Externalizable is roughly an old-java version of Serializable.  Does that class
implement both interfaces?  Can you take away the Externalizable interface if it's being used?


On Mon, Dec 2, 2013 at 7:15 PM, Matt Cheah <mcheah@palantir.com<mailto:mcheah@palantir.com>>
wrote:
Hi everyone,

I'm running into a case where I'm creating a Java RDD of an Externalizable class, and getting
this stack trace:

java.io.InvalidClassException (java.io.InvalidClassException: com.palantir.finance.datatable.server.spark.WritableDataRow;
Serializable incompatible with Externalizable)
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634)java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)java.io.ObjectInputStream.readClass(ObjectInputStream.java:1483)<some
other Java stuff>java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
I'm running on a spark cluster generated by the EC2 Scripts. This doesn't happen if I'm running
things with local[N]. Any ideas?
Thanks,
-Matt Cheah


Mime
View raw message