ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Exceeding the DataStorageConfiguration#getMaxWalArchiveSize due to historical rebalance
Date Tue, 04 May 2021 15:43:01 GMT
Hello!

Maybe we can have a mechanic here similar (or equal) to checkpoint based
write throttling?

So we will be throttling for both checkpoint page buffer and WAL limit.

Regards,
-- 
Ilya Kasnacheev


вт, 4 мая 2021 г. в 11:29, ткаленко кирилл <tkalkirill@yandex.ru>:

> Hello everybody!
>
> At the moment, if there are partitions for the rebalance for which the
> historical rebalance will be used, then we reserve segments in the WAL
> archive (we do not allow cleaning the WAL archive) until the rebalance for
> all cache groups is over.
>
> If a cluster is under load during the rebalance, WAL archive size may
> significantly exceed limits set in
> DataStorageConfiguration#getMaxWalArchiveSize until the process is
> complete. This may lead to user issues and nodes may crash with the "No
> space left on device" error.
>
> We have a system property IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE by
> default 0.5, which sets the threshold (multiplied by getMaxWalArchiveSize)
> from which and up to which the WAL archive will be cleared, i.e. sets the
> size of the WAL archive that will always be on the node. I propose to
> replace this system property with the
>  DataStorageConfiguration#getWalArchiveSize in bytes, the default is
> (getMaxWalArchiveSize * 0.5) as it is now.
>
> Main proposal:
> When theDataStorageConfiguration#getMaxWalArchiveSize is reached, cancel
> and do not give the reservation of the WAL segments until we reach
> DataStorageConfiguration#getWalArchiveSize. In this case, if there is no
> segment for historical rebalance, we will automatically switch to full
> rebalance.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message