spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Baretta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5235) Determine serializability of SQLContext
Date Wed, 14 Jan 2015 19:48:35 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277555#comment-14277555
] 

Alex Baretta commented on SPARK-5235:
-------------------------------------

[~rxin] Could be. All I'm saying is that your change was not intended to make SQLContext not
Serializable. Even if we all agree that it would be cleaner, "taking the opportunity" offered
by this regression to remove the Serializable trait from SQLContext is not a good idea, as
there is no emergency here. Printing a warning, writing something in docs of version 1.3 and
then waiting until 1.4 would be a better process. 

> Determine serializability of SQLContext
> ---------------------------------------
>
>                 Key: SPARK-5235
>                 URL: https://issues.apache.org/jira/browse/SPARK-5235
>             Project: Spark
>          Issue Type: Sub-task
>            Reporter: Alex Baretta
>
> The SQLConf field in SQLContext is neither Serializable nor transient. Here's the stack
trace I get when running SQL queries against a Parquet file.
> {code}
> Exception in thread "Thread-43" org.apache.spark.SparkException: Job aborted due to stage
failure: Task not serializable: java.io.NotSerializableException: org.apache.spark.sql.SQLConf
>         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1195)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1184)
>         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1183)
>         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1183)
>         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:843)
>         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:779)
>         at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:763)
>         at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1364)
>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>         at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1356)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>         at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>         at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message