flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8876) Improve concurrent access handling in stateful serializers
Date Tue, 06 Mar 2018 12:54:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387708#comment-16387708
] 

ASF GitHub Bot commented on FLINK-8876:
---------------------------------------

Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/5645
  
    We got a bunch of failing tests:
    ```
    Failed tests: 
      MultidimensionalArraySerializerTest.testObjectArrays:84 Exception in test: Serializer
already accessed by thread main
      PojoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testBeanStyleObjects:95->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      PojoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testCompositeObject:75->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      PojoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedInterfaces:124->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      PojoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedObjects:85->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      PojoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testSimpleTypesObjects:64->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      PojoSubclassSerializerTest>SerializerTestBase.testSerializedCopyAsSequence:402 Exception
in test: Serializer already accessed by thread main
      PojoSubclassSerializerTest>SerializerTestBase.testSerializedCopyIndividually:364
Exception in test: Serializer already accessed by thread main
      SubclassFromInterfaceSerializerTest>SerializerTestBase.testSerializedCopyAsSequence:402
Exception in test: Serializer already accessed by thread main
      SubclassFromInterfaceSerializerTest>SerializerTestBase.testSerializedCopyIndividually:364
Exception in test: Serializer already accessed by thread main
      TupleSerializerTest.testTuple5CustomObjects:215->runTests:229 Exception in test:
Serializer already accessed by thread main
      KryoGenericArraySerializerTest>AbstractGenericArraySerializerTest.testBeanStyleObjects:120->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      KryoGenericArraySerializerTest>AbstractGenericArraySerializerTest.testCompositeObject:93->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      KryoGenericArraySerializerTest>AbstractGenericArraySerializerTest.testNestedObjects:103->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      KryoGenericArraySerializerTest>AbstractGenericArraySerializerTest.testSimpleTypesObjects:82->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      KryoGenericArraySerializerTest>AbstractGenericArraySerializerTest.testString:63->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testBeanStyleObjects:95->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testCompositeObject:75->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest.testJavaDequeue:68->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest.testJavaList:50->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest.testJavaSet:59->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedInterfaces:124->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedObjects:85->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testSimpleTypesObjects:64->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testString:41->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoSerializerClassLoadingTest>SerializerTestBase.testSerializedCopyAsSequence:402
Exception in test: Serializer already accessed by thread main
      KryoSerializerClassLoadingTest>SerializerTestBase.testSerializedCopyIndividually:364
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testBeanStyleObjects:95->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testCompositeObject:75->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest.testJodaTime:48->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testNestedInterfaces:124->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testNestedObjects:85->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testSimpleTypesObjects:64->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      KryoWithCustomSerializersTest>AbstractGenericTypeSerializerTest.testString:41->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
    ```
    
    ```
    Failed tests: 
      AvroGenericArraySerializerTest>AbstractGenericArraySerializerTest.testBeanStyleObjects:120->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      AvroGenericArraySerializerTest>AbstractGenericArraySerializerTest.testCompositeObject:93->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      AvroGenericArraySerializerTest>AbstractGenericArraySerializerTest.testNestedObjects:103->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      AvroGenericArraySerializerTest>AbstractGenericArraySerializerTest.testSimpleTypesObjects:82->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      AvroGenericArraySerializerTest>AbstractGenericArraySerializerTest.testString:63->AbstractGenericArraySerializerTest.runTests:152->AbstractGenericArraySerializerTest.runTests:173
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testBeanStyleObjects:95->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testCompositeObject:75->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedInterfaces:124->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testNestedObjects:85->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testSimpleTypesObjects:64->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroGenericTypeSerializerTest>AbstractGenericTypeSerializerTest.testString:41->AbstractGenericTypeSerializerTest.runTests:155
Exception in test: Serializer already accessed by thread main
      AvroSerializerEmptyArrayTest.testBookSerialization:42 Exception in test: Serializer
already accessed by thread main
      AvroSerializerEmptyArrayTest.testSerialization:67 Exception in test: Serializer already
accessed by thread main
      AvroSerializerTest>SerializerTestBase.testSerializedCopyAsSequence:402 Exception
in test: Serializer already accessed by thread main
      AvroSerializerTest>SerializerTestBase.testSerializedCopyIndividually:364 Exception
in test: Serializer already accessed by thread main
    ```
    
    The changes itself look(ed) good to me, but we'll have to check whether the tests do something
wrong or the code in the PR uses wrong assumptions.
    
    I've verified manually that the `CONCURRENT_ACCESS_CHECK` setup works properly outside
of tests. I'm wondering whether we can test this as well in `flink-end-to-end-tests`.


> Improve concurrent access handling in stateful serializers
> ----------------------------------------------------------
>
>                 Key: FLINK-8876
>                 URL: https://issues.apache.org/jira/browse/FLINK-8876
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>            Priority: Major
>             Fix For: 1.5.0, 1.6.0
>
>
> Some stateful serializers produce incorrect results when accidentally accessed by multiple
threads concurrently.
>  To better catch these cases, I suggest to add concurrency checks that are active only
when debug logging is enabled, and during test runs.
> This is inspired by Kryo's checks for concurrent access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message