cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Borgström (Commented) (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3466) Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
Date Tue, 08 Nov 2011 08:42:51 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146165#comment-13146165
] 

Jonas Borgström commented on CASSANDRA-3466:
--------------------------------------------

> Do you observe the same behavior if both nodes start life as 1.x?

I spoke too soon. I did some more testing with fresh 1.0.2 installs (and empty data directories).
{code}
1. Start node1 and node2 and create keyspace and cf.
2. Stop node2, wait until "is now dead".
3. Trigger hint using "set bar[x][x]='x';"
4. Start node2, wait for hint delivery... Nothing happens
5. Restart node1, hints are finally delivered.
{code}
Now it gets really weird...
{code}
2. Stop node2, wait until "is now dead".
3. Trigger hint using "set bar[x][x]='x';"
4. Start node2. Node1 attempts to deliver hints but fails with:
 INFO 09:29:35,153 Started hinted handoff for token: 23495828435496583962471242736585511198
with IP: /127.0.0.2
ERROR 09:29:35,212 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.AssertionError
	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
	at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
	at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:680)

On node1:
[default@system] list HintsColumnFamily;
Using default limit of 100
-------------------
RowKey: 11ad21c7d112e903d71645d1d951911e
=> (super_column=b2cd983009e311e10000fe8ebeead9cf,
     (column=6b6579, value=78, timestamp=1320740935987, ttl=864000)
     (column=6d75746174696f6e, value=0003666f6f00017800000001000003e801000003e880000000800000000000000000000001000178000004b134f65401d80000000139,
timestamp=1320740935987, ttl=864000)
     (column=7461626c65, value=666f6f, timestamp=1320740935987, ttl=864000)
     (column=76657273696f6e, value=00000003, timestamp=1320740935987, ttl=864000))
=> (super_column=f929a36009e211e10000fe8ebeead9df,
     (column=6b6579, value=78, timestamp=1320740624538, ttl=864000)
     (column=7461626c65, value=666f6f, timestamp=1320740624538, ttl=864000))
1 Row Returned.
{code}

Is that a 0.8 hint showing up out of the blue on a pure 1.0.2 cluster? Or did I mess up my
test somehow?

                
> Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-3466
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3466
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.2
>            Reporter: Jonas Borgström
>              Labels: hintedhandoff
>
> While testing rolling upgrades from 0.8.7 to 1.0.2 on a test cluster I've noticed that
hinted hand-off didn't always work properly. Hints generated on an upgraded node does not
seem to be delivered to other newly upgraded nodes once they rejoin the ring. They only way
I've found to get a node to deliver its hints is to restart it.
> Here's some steps to reproduce this issue:
> 1. Install cassandra 0.8.7 on node1 and node2 using default settings.
> 2. Create keyspace foo with {replication_factor: 2}. Create column family bar
> 3. Shutdown node2 
> 4. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 5. Start node2 and verify that hinted handoff is performed and HintsColumnFamily becomes
empty again.
> 6. Upgrade and restart node1
> 7. Shutdown node2 
> 8. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 9. Upgrade and start node2
> 10. Notice that hinted handoff is *not* performed when "node2" comes back. (Only if node1
is restarted)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message