cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Boudreault (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10156) Creating Materialized views concurrently leads to missing data
Date Sat, 22 Aug 2015 17:21:45 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708056#comment-14708056
] 

Alan Boudreault edited comment on CASSANDRA-10156 at 8/22/15 5:21 PM:
----------------------------------------------------------------------

This an output example of the script:
{code}
(cassandra3_env)2.2.2 ~/Desktop
❯ sh mv_test.sh
creating 3 node c*3.0 cluster
http://git-wip-us.apache.org/repos/asf/cassandra.git git:cassandra-3.0
Fetching Cassandra updates...
Current cluster is now: mv_test
[node1 ERROR] objc[65819]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
[node3 ERROR] objc[65818]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
[node2 ERROR] objc[65817]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.


creating keyspace and base table with 5000 rows of data


adding materialized views


napping while the work gets done


cluster schema in sync?
objc[65939]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]


objc[65961]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]


objc[65983]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]




checking mv system tables on node1

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking mv system tables on node2

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking mv system tables on node3

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking counts on table and mvs -- all should be 5000
Consistency level set to ALL.

 count
-------
  5000

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  2012

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  1807

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  3625

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  5000

(1 rows)

Warnings :
Aggregation query used without partition key

COMPLETED
{code}


was (Author: aboudreault):
This a script output example:
{code}
(cassandra3_env)2.2.2 ~/Desktop
❯ sh mv_test.sh
creating 3 node c*3.0 cluster
http://git-wip-us.apache.org/repos/asf/cassandra.git git:cassandra-3.0
Fetching Cassandra updates...
Current cluster is now: mv_test
[node1 ERROR] objc[65819]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
[node3 ERROR] objc[65818]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
[node2 ERROR] objc[65817]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.


creating keyspace and base table with 5000 rows of data


adding materialized views


napping while the work gets done


cluster schema in sync?
objc[65939]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]


objc[65961]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]


objc[65983]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/bin/java
and /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.

Cluster Information:
    Name: mv_test
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        97154a8b-4863-347e-b3d4-ca8ffcf7884b: [127.0.0.1, 127.0.0.2, 127.0.0.3]




checking mv system tables on node1

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking mv system tables on node2

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking mv system tables on node3

 keyspace_name | view_name | generation_number | last_token
---------------+-----------+-------------------+------------

(0 rows)

 keyspace_name | view_name
---------------+-----------
  lots_o_views | src_by_c1
  lots_o_views | src_by_c2
  lots_o_views | src_by_c3
  lots_o_views | src_by_c4

(4 rows)


checking counts on table and mvs -- all should be 5000
Consistency level set to ALL.

 count
-------
  5000

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  2012

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  1807

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  3625

(1 rows)

Warnings :
Aggregation query used without partition key


 count
-------
  5000

(1 rows)

Warnings :
Aggregation query used without partition key

COMPLETED
{code}

> Creating Materialized views concurrently leads to missing data
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-10156
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10156
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alan Boudreault
>             Fix For: 3.x
>
>         Attachments: mv_test_bad.sh, mv_test_good.sh
>
>
> [~nutbunnies] was writing dtests that create multiple tables concurrently. He also wrote
a test that creates multiple MV but has not been able to get it works properly. After some
debugging outside of dtest, it seems that there is an issue if we create more than 1 MV at
the same time. There is no errors in the log but the MV are never entirely populated and are
missing data.
> I've attached 2 scripts:
> [^mv_test_bad.sh]: is the one that reproduce the issue. It creates 4 MVs at the same
time. At the end, some data are missing in the MVs and there is nothing in system.hints or
system.batchlog.
> [^mv_test_good.sh]: is the same script but that waits 10 seconds between each MV creation,
which results in 4 MVs with all the data.
> Some more notes from Andrew:
> {code}
> - lowering the number of rows inserted below ~1000 won't exhibit the inconsistent behavior
> - adding more columns/MV make it worse -- more of the MVs counts are consistently wrong
> - multiple runs will range in disagreement -- usually one of the MVs is correct though
> - the describe cluster and system.mv* queries always "look" good
> {code}
> Thanks Andrew for finding this bug! 
> //cc [~carlyeks] [~tjake] [~enigmacurry]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message