spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5506) java.lang.ClassCastException using lambda expressions in combination of spark and Servlet
Date Sat, 31 Jan 2015 10:24:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299750#comment-14299750
] 

Sean Owen commented on SPARK-5506:
----------------------------------

Hm, why would it relate to servlets in particular, I wonder? what else did you try it with?
I wonder if it can be narrowed down. Like what if the outer lambda is replaced?

> java.lang.ClassCastException using lambda expressions in combination of spark and Servlet
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-5506
>                 URL: https://issues.apache.org/jira/browse/SPARK-5506
>             Project: Spark
>          Issue Type: Question
>          Components: Spark Core
>    Affects Versions: 1.2.0
>         Environment: spark server: Ubuntu 14.04 amd64
> $ java -version
> java version "1.8.0_25"
> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
>            Reporter: Milad Khajavi
>
> I'm trying to build a web API for my Apache spark jobs using sparkjava.com framework.
My code is:
> @Override
> public void init() {
>     get("/hello",
>             (req, res) -> {
>                 String sourcePath = "hdfs://spark:54310/input/*";
>                 SparkConf conf = new SparkConf().setAppName("LineCount");
>                 conf.setJars(new String[] { "/home/sam/resin-4.0.42/webapps/test.war"
});
>                 File configFile = new File("config.properties");
>                 String sparkURI = "spark://hamrah:7077";
>                 conf.setMaster(sparkURI);
>                 conf.set("spark.driver.allowMultipleContexts", "true");
>                 JavaSparkContext sc = new JavaSparkContext(conf);
>                 @SuppressWarnings("resource")
>                 JavaRDD<String> log = sc.textFile(sourcePath);
>                 JavaRDD<String> lines = log.filter(x -> {
>                     return true;
>                 });
>                 return lines.count();
>             });
> }
> If I remove the lambda expression or put it inside a simple jar rather than a web service
(somehow a Servlet) it will run without any error. But using a lambda expression inside a
Servlet will result this exception:
> 15/01/28 10:36:33 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, hamrah): java.lang.ClassCastException:
cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDD$$anonfun$filter$1.f$1
of type org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaRDD$$anonfun$filter$1
> at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)
> at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1999)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> P.S: I tried combination of jersey and javaspark with jetty, tomcat and resin and all
of them led me to the same result.
> Here the same issue: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-YARN-java-lang-ClassCastException-SerializedLambda-to-org-apache-spark-api-java-function-Fu1-tt21261.html
> This is my colleague question in stackoverflow: http://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message