spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody Koeninger (JIRA)" <>
Subject [jira] [Commented] (SPARK-17829) Stable format for offset log
Date Thu, 27 Oct 2016 02:27:58 GMT


Cody Koeninger commented on SPARK-17829:

I'm not telling you to do it that way, just asking if you had considered it.  General advantage
of typeclasses being separating concerns (should all these classes need to know about json)
and getting inductive definitions for free (if you have a serializer for container, you have
a serializer for any container of nested serializable). If all the stuff you're looking at
modifying already knows about java serialization it may not be a big deal though.

Specifically about the using a seq instead of array for compactible file stream, isn't there
an existing warning in the code as to why that's using an array, due to pathological behavior
on large linked lists?

> Stable format for offset log
> ----------------------------
>                 Key: SPARK-17829
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Michael Armbrust
>            Assignee: Tyson Condie
> Currently we use java serialization for the WAL that stores the offsets contained in
each batch.  This has two main issues:
>  - It can break across spark releases (though this is not the only thing preventing us
from upgrading a running query)
>  - It is unnecessarily opaque to the user.
> I'd propose we require offsets to provide a user readable serialization and use that
instead.  JSON is probably a good option.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message