spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-5506) java.lang.ClassCastException using lambda expressions in combination of spark and Servlet
Date Sat, 31 Jan 2015 09:20:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-5506:
-----------------------------
    Priority: Major  (was: Blocker)

(In case anyone else gets confused for a second, there is another Java project called "Spark",
and it is also being used here!)

Interesting, all I can glean from this is that something did not deserialize the lambda expression
passed to filter(). Given that it only occurs inside of this web framework, is it not possible
it is somehow failing to handle the lambda? Because it indeed works correctly in your example
and in unit tests.

> java.lang.ClassCastException using lambda expressions in combination of spark and Servlet
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-5506
>                 URL: https://issues.apache.org/jira/browse/SPARK-5506
>             Project: Spark
>          Issue Type: Question
>          Components: Spark Core
>    Affects Versions: 1.2.0
>         Environment: spark server: Ubuntu 14.04 amd64
> $ java -version
> java version "1.8.0_25"
> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
>            Reporter: Milad Khajavi
>
> I'm trying to build a web API for my Apache spark jobs using sparkjava.com framework.
My code is:
> @Override
> public void init() {
>     get("/hello",
>             (req, res) -> {
>                 String sourcePath = "hdfs://spark:54310/input/*";
>                 SparkConf conf = new SparkConf().setAppName("LineCount");
>                 conf.setJars(new String[] { "/home/sam/resin-4.0.42/webapps/test.war"
});
>                 File configFile = new File("config.properties");
>                 String sparkURI = "spark://hamrah:7077";
>                 conf.setMaster(sparkURI);
>                 conf.set("spark.driver.allowMultipleContexts", "true");
>                 JavaSparkContext sc = new JavaSparkContext(conf);
>                 @SuppressWarnings("resource")
>                 JavaRDD<String> log = sc.textFile(sourcePath);
>                 JavaRDD<String> lines = log.filter(x -> {
>                     return true;
>                 });
>                 return lines.count();
>             });
> }
> If I remove the lambda expression or put it inside a simple jar rather than a web service
(somehow a Servlet) it will run without any error. But using a lambda expression inside a
Servlet will result this exception:
> 15/01/28 10:36:33 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, hamrah): java.lang.ClassCastException:
cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDD$$anonfun$filter$1.f$1
of type org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaRDD$$anonfun$filter$1
> at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)
> at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1999)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> P.S: I tried combination of jersey and javaspark with jetty, tomcat and resin and all
of them led me to the same result.
> Here the same issue: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-YARN-java-lang-ClassCastException-SerializedLambda-to-org-apache-spark-api-java-function-Fu1-tt21261.html
> This is my colleague question in stackoverflow: http://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message