flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9377) Remove writing serializers as part of the checkpoint meta information
Date Mon, 08 Oct 2018 11:11:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641671#comment-16641671

ASF GitHub Bot commented on FLINK-9377:

dawidwys commented on a change in pull request #6711: [FLINK-9377] [core, state backends]
Remove serializers from checkpoints
URL: https://github.com/apache/flink/pull/6711#discussion_r223313725

 File path: flink-core/src/test/java/org/apache/flink/api/common/typeutils/TypeSerializerSerializationUtilTest.java
 @@ -182,15 +178,17 @@ public void testSerializeConfigurationSnapshots() throws Exception
 	public void testFailsWhenConfigurationSnapshotClassNotFound() throws Exception {
 		byte[] serializedConfig;
 		try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
-			TypeSerializerSerializationUtil.writeSerializerConfigSnapshot(
-				new DataOutputViewStreamWrapper(out), new TypeSerializerSerializationUtilTest.TestConfigSnapshot(123,
+			TypeSerializerSnapshotSerializationUtil.writeSerializerSnapshot(
+				new DataOutputViewStreamWrapper(out),
+				new TypeSerializerSerializationUtilTest.TestConfigSnapshot<>(123, "foobar"),
+				StringSerializer.INSTANCE);
 			serializedConfig = out.toByteArray();
 		try (ByteArrayInputStream in = new ByteArrayInputStream(serializedConfig)) {
 			// read using a dummy classloader
-			TypeSerializerSerializationUtil.readSerializerConfigSnapshot(
-				new DataInputViewStreamWrapper(in), new URLClassLoader(new URL[0], null));
+			TypeSerializerSnapshotSerializationUtil.readSerializerSnapshot(
+				new DataInputViewStreamWrapper(in), new URLClassLoader(new URL[0], null), null);
 			fail("Expected a ClassNotFoundException wrapped in IOException");
 Review comment:
   Could we move the combination of `fail` and `catch` to `@Test(expected=IOException.class)`?

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Remove writing serializers as part of the checkpoint meta information
> ---------------------------------------------------------------------
>                 Key: FLINK-9377
>                 URL: https://issues.apache.org/jira/browse/FLINK-9377
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.7.0
> When writing meta information of a state in savepoints, we currently write both the state
serializer as well as the state serializer's configuration snapshot.
> Writing both is actually redundant, as most of the time they have identical information.
>  Moreover, the fact that we use Java serialization to write the serializer and rely on
it to be re-readable on the restore run, already poses problems for serializers such as the
{{AvroSerializer}} (see discussion in FLINK-9202) to perform even a compatible upgrade.
> The proposal here is to leave only the config snapshot as meta information, and use that
as the single source of truth of information about the schema of serialized state.
>  The config snapshot should be treated as a factory (or provided to a factory) to re-create
serializers capable of reading old, serialized state.

This message was sent by Atlassian JIRA

View raw message