spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Commented] (SPARK-4737) Prevent serialization errors from ever crashing the DAG scheduler
Date Sat, 10 Jan 2015 01:32:34 GMT


Patrick Wendell commented on SPARK-4737:

It's great to see this go in. Thanks [~mcheah]!

> Prevent serialization errors from ever crashing the DAG scheduler
> -----------------------------------------------------------------
>                 Key: SPARK-4737
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.2, 1.1.1, 1.2.0
>            Reporter: Patrick Wendell
>            Assignee: Matthew Cheah
>            Priority: Blocker
>             Fix For: 1.3.0
> Currently in Spark we assume that when tasks are serialized in the TaskSetManager that
the serialization cannot fail. We assume this because upstream in the DAGScheduler we attempt
to catch any serialization errors by serializing a single partition. However, in some cases
this upstream test is not accurate - i.e. an RDD can have one partition that can serialize
cleanly but not others.
> Do do this in the proper way we need to catch and propagate the exception at the time
of serialization. The tricky bit is making sure it gets propagated in the right way.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message