kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adar Lieber-Dembo <a...@cloudera.com>
Subject Re: Service unavailable due to exceeding transaction memory consumption
Date Fri, 22 Feb 2019 18:49:19 GMT
The error indicates that the set of outstanding transactions on this
tablet is hovering around 64MB. The error message is inherently racy
in that the values included aren't necessarily the values that
triggered the failure, so don't worry about that aspect of it.

A couple things to think about:
1. Normally the Kudu client retries a write if it receives a "service
unavailable" error. Does the Spark error indicate that retrying took
place? Was there an eventual timeout?
2. It's pretty unusual to be up against this 64M limit; it's usually a
sign that your write batches may be too large, or that the tablet
leader isn't able to replicate to its followers. What's this workload
look like? What is being inserted? How large are the rows? How many
rows per batch? What are your non-default configs? Does 'kudu cluster
ksck' come back healthy?
3. If you want to know more about the transaction state of a tserver
at any given time, visit the tserver's /transactions page in its web
UI (linked via "Dashboards").

On Fri, Feb 22, 2019 at 5:34 AM Nabeelah Harris
<nabeelah.harris@impact.com> wrote:
>
> Hi there
>
> While writing to a particular partition of a Kudu table using
> KuduContext.insertRows, I receive the following error: "Service
> unavailable: Transaction failed, tablet <tabletId> transaction memory
> consumption (67075227) has exceeded its limit (67108864) or the limit
> of an ancestral tracker". How would this occur is the consumption is
> less than the limit? Writing to other partitions seems to work just
> fine.
>
> I am inclined to believe that this isn't due to an issue with the
> tablet or tablet server itself, as I get the same error when I drop
> and re-create the partition, and the partition ends up on a different
> tablet/server entirely. Is this simply because I'm trying to write too
> much data at once? What are the kinds of logs I can look for on the
> masters/tablet servers to indicate that this might happen? What are
> some of the configs I might be able to tweak regarding this issue?
>
> Thanks
> Nabeelah

Mime
View raw message