Ok, I found the mistake!. Wow, it really come to me by inspiration, if not I don't know how this suddenly come to me.

The problem is that I packed my application in a jar file with the old "spark-assembly-0.8.0-incubating-hadoop1.0.4.jar" in it. So that is the reason why workers and master have different versions of the file. I re-packeted it and run it just fine!.

Thanks for all the help,

On Thu, Oct 3, 2013 at 4:05 PM, Eduardo Berrocal <eberroca@hawk.iit.edu> wrote:
And by .java, I mean .scala

On Thu, Oct 3, 2013 at 4:03 PM, Eduardo Berrocal <eberroca@hawk.iit.edu> wrote:
That directory is not present on version 0.8.0 (the one I am using).

However, there are the files for FlatMapFunction in spark 0.8.0:

$ find ./ -name FlatMapFunction*

All of the FlatMapFunction.class files have a modification time bigger than the .java file, which I suspect means that they have been created from the new version of the java file (all of them!)


On Thu, Oct 3, 2013 at 3:38 PM, Martin Weindel <martin.weindel@gmail.com> wrote:
Hi Eduardo,

if you are using Spark 0.7.3, I remember that I had to replace the class file additional at


Am 03.10.2013 22:35, schrieb Eduardo Berrocal:
Hi Martin,

Yes, that is what is seems. However, it is unlikely that is the case, because I have all spark classes on my home, which is mounted on NFS to all nodes. Unless there is something else I am missing...


On Thu, Oct 3, 2013 at 3:29 PM, Martin Weindel <martin.weindel@gmail.com> wrote:
Hi Eduardo,

it seems to me that your second problem is caused by inconsistent, i.e. different classes in master and worker JVMs.
Are you sure, that you have replaced the changed FlatMapFunction on all worker nodes and also on master?


13/10/03 13:27:44 INFO cluster.ClusterTaskSetManager: Lost TID 0 (task
13/10/03 13:27:44 INFO cluster.ClusterTaskSetManager: Loss was due to
org.apache.spark.api.java.function.FlatMapFunction; local class
incompatible: stream classdesc serialVersionUID = -1748278142466443391,
local class serialVersionUID = 2220150375729402137