cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh McKenzie (Jira)" <>
Subject [jira] [Updated] (CASSANDRA-16663) Request-Based Native Transport Rate-Limiting
Date Tue, 03 Aug 2021 17:33:00 GMT


Josh McKenzie updated CASSANDRA-16663:
    Reviewers: Benedict Elliott Smith, Josh McKenzie  (was: Benedict Elliott Smith, Sam Tunnicliffe)

> Request-Based Native Transport Rate-Limiting
> --------------------------------------------
>                 Key: CASSANDRA-16663
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Messaging/Client
>            Reporter: Caleb Rackliffe
>            Assignee: Caleb Rackliffe
>            Priority: Normal
>             Fix For: 4.x
>          Time Spent: 6h 10m
>  Remaining Estimate: 0h
> Together, CASSANDRA-14855, CASSANDRA-15013, and CASSANDRA-15519 added support for a runtime-configurable,
per-coordinator limit on the number of bytes allocated for concurrent requests over the native
protocol. It supports channel back-pressure by default, and optionally supports throwing OverloadedException
if that is requested in the relevant connection’s STARTUP message.
> This can be an effective tool to prevent the coordinator from running out of memory,
but it may not correspond to how expensive a queries are or provide a direct conceptual mapping
to how users think about request capacity. I propose adding the option of request-based (or
perhaps more correctly message-based) back-pressure, coexisting with (and reusing the logic
that supports) the current bytes-based back-pressure.
> _We can roll this forward in phases_, where the server’s cost accounting becomes more
accurate, we segment limits by operation type/keyspace/etc., and the client/driver reacts
more intelligently to (especially non-back-pressure) overload, _but something minimally viable
could look like this_:
> 1.) Reuse most of the existing logic in Limits, et al. to support a simple per-coordinator
limit only on native transport requests per second. Under this limit will be CQL reads and
writes, but also auth requests, prepare requests, and batches. This is obviously simplistic,
and it does not account for the variation in cost between individual queries, but even a fixed
cost model should be useful in aggregate.
>  * If the client specifies THROW_ON_OVERLOAD in its STARTUP message at connection time,
a breach of the per-node limit will result in an OverloadedException being propagated to the
client, and the server will discard the request.
>  * If THROW_ON_OVERLOAD is not specified, the server will stop consuming messages from
the channel/socket, which should back-pressure the client, while the message continues to
be processed.
> 2.) This limit is infinite by default (or simply disabled), and can be enabled via the
YAML config or JMX at runtime. (It might be cleaner to have a no-op rate limiter that's used
when the feature is disabled entirely.)
> 3.) The current value of the limit is available via JMX, and metrics around coordinator
operations/second are already available to compare against it.
> 4.) Any interaction with existing byte-based limits will intersect. (i.e. A breach of
any limit, bytes or request-based, will actuate back-pressure or OverloadedExceptions.)
> In this first pass, explicitly out of scope would be any work on the client/driver side.
> In terms of validation/testing, our biggest concern with anything that adds overhead
on a very hot path is performance. In particular, we want to fully understand how the client
and server perform along two axes constituting 4 scenarios. Those are a.) whether or not we
are breaching the request limit and b.) whether the server is throwing on overload at the
behest of the client. Having said that, query execution should dwarf the cost of limit accounting.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message