cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Spriegel (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7886) Coordinator should not wait for read timeouts when replicas hit Exceptions
Date Fri, 02 Jan 2015 15:22:14 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262943#comment-14262943
] 

Christian Spriegel edited comment on CASSANDRA-7886 at 1/2/15 3:22 PM:
-----------------------------------------------------------------------

Hi [~thobbs],

uploaded new patch: V6

Here is what I did:
- Fixed logging of TOEs...
-- ... in StorageProxy for local reads
-- ... in MessageDeliveryTask for remote reads
- Added partitionKey(as DecoratedKey) and lastCellName logging to TOE.
- Changed SliceQueryFilter not to throw TOEs Exception for System-keyspace. Cassandra does
not seem to like TOEs in system queries. These TOEs will always be logged as warnings instead.


This is how TOEs look like in system.log:
{code}
ERROR [SharedPool-Worker-1] 2015-01-02 15:07:24,878 MessageDeliveryTask.java:81 - Scanned
over 201 tombstones in test.test; 100 columns were requested; query aborted (see tombstone_failure_threshold);
partitionKey=DecoratedKey(78703492656118554854272571946195123045, 31); lastCell=188; delInfo={deletedAt=-9223372036854775808,
localDeletion=2147483647}; slices=[-]
{code}

kind regards,
Christian



was (Author: christianmovi):
Hi [~thobbs],

uploaded new patch: V6

Here is what I did:
- Fixed logging of TOEs...
-- ... in StorageProxy for local reads
-- ... in MessageDeliveryTask for remote reads
- Added partitionKey(as DecoratedKey) and lastCellName logging to TOE.
- Changed SliceQueryFilter not to throw TOEs Exception for System-keyspace. Cassandra does
not seem to like TOEs in system queries. These TOEs will always be logged as warnings instead.


This is how TOEs look like in system.log:
{quote}
ERROR [SharedPool-Worker-1] 2015-01-02 15:07:24,878 MessageDeliveryTask.java:81 - Scanned
over 201 tombstones in test.test; 100 columns were requested; query aborted (see tombstone_failure_threshold);
partitionKey=DecoratedKey(78703492656118554854272571946195123045, 31); lastCell=188; delInfo={deletedAt=-9223372036854775808,
localDeletion=2147483647}; slices=[-]
{quote}

kind regards,
Christian


> Coordinator should not wait for read timeouts when replicas hit Exceptions
> --------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7886
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7886
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: Tested with Cassandra 2.0.8
>            Reporter: Christian Spriegel
>            Assignee: Christian Spriegel
>            Priority: Minor
>              Labels: protocolv4
>             Fix For: 3.0
>
>         Attachments: 7886_v1.txt, 7886_v2_trunk.txt, 7886_v3_trunk.txt, 7886_v4_trunk.txt,
7886_v5_trunk.txt, 7886_v6_trunk.txt
>
>
> *Issue*
> When you have TombstoneOverwhelmingExceptions occuring in queries, this will cause the
query to be simply dropped on every data-node, but no response is sent back to the coordinator.
Instead the coordinator waits for the specified read_request_timeout_in_ms.
> On the application side this can cause memory issues, since the application is waiting
for the timeout interval for every request.Therefore, if our application runs into TombstoneOverwhelmingExceptions,
then (sooner or later) our entire application cluster goes down :-(
> *Proposed solution*
> I think the data nodes should send a error message to the coordinator when they run into
a TombstoneOverwhelmingException. Then the coordinator does not have to wait for the timeout-interval.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message