ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ткаленко кирилл <tkalkir...@yandex.ru>
Subject Re: [DISCUSSION] Cache warmup
Date Tue, 04 Aug 2020 10:42:44 GMT
Hi, Denis!

Previously, I answered Slava about implementation that I keep in mind, now it will be possible
to add own warm-up strategy implementations. Which will be possible to implement in different
ways. 

At the moment, I suggest implementing one "Load all" strategy, which will be effective if
persistent storage is less than RAM.


28.07.2020, 19:46, "Denis Mekhanikov" <dmekhanikov@gmail.com>:
> Kirill,
>
> That will be a great feature! Other popular databases already have it (e.g.
> Postgres: https://www.postgresql.org/docs/11/pgprewarm.html), so it's good
> that we're also going to have it in Ignite.
>
> What implementation of CacheWarmup interface do you have in mind? Will
> there be some preconfigured implementation, and will users be able to
> implement it themselves?
>
> Do you think it should be cache-based? I would say that a DataRegion-based
> warm-up would come more naturally. Page IDs that are loaded into the data
> region can be dumped periodically to disk and recovered on restarts. This
> is more or less how it works in Postgres.
> I'm afraid that if we make it cache-based, the implementation won't be that
> obvious. We already have an API for warmup that appeared to be pretty much
> impossible to apply in a useful way:
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#preloadPartition-int-
> Let's make sure that our new tool for warming up is actually useful.
>
> Denis
>
> вт, 28 июл. 2020 г. в 09:17, Zhenya Stanilovsky <arzamas123@mail.ru.invalid
>> :
>
>>  Looks like we need additional func for static caches, for
>>  example: warmup(List<CacheConfiguration> cconf) it would be helpful for
>>  spring too.
>>
>>  >
>>  >------- Forwarded message -------
>>  >From: "Вячеслав Коптилин" < slava.koptilin@gmail.com >
>>  >To: dev@ignite.apache.org
>>  >Cc:
>>  >Subject: Re: [DISCUSSION] Cache warmup
>>  >Date: Mon, 27 Jul 2020 16:47:48 +0300
>>  >
>>  >Hello Kirill,
>>  >
>>  >Thanks a lot for driving this activity. If I am not mistaken, this
>>  >discussion relates to IEP-40.
>>  >
>>  >> I suggest adding a warmup phase after recovery here [1] after [2],
>>  before
>>  >discovery.
>>  >This means that the user's thread, which starts Ignite via
>>  >Ignition.start(), will wait for ana additional step - cache warm-up.
>>  >I think this fact has to be clearly mentioned in our documentation (at
>>  >Javadocat least) because this step can be time-consuming.
>>  >
>>  >> I suggest adding a new interface:
>>  >I would change it a bit. First of all, it would be nice to place this
>>  >interface to a public package and get rid of using GridCacheContext,
>>  >which is an internal class and it should not leak to the public API in any
>>  >case.
>>  >Perhaps, this parameter is not needed at all or we should add some public
>>  >abstraction instead of internal class.
>>  >
>>  >package org.apache.ignite.configuration;
>>  >
>>  >import org.apache.ignite.IgniteCheckedException;
>>  >import org.apache.ignite.lang.IgniteFuture;
>>  >
>>  >public interface CacheWarmupper {
>>  > /**
>>  > * Warmup cache.
>>  > *
>>  > * @param cachename Cache name.
>>  > * @return Future cache warmup.
>>  > * @throws IgniteCheckedException If failed.
>>  > */
>>  > IgniteFuture<?> warmup(String cachename) throws
>>  >IgniteCheckedException;
>>  >}
>>  >
>>  >Thanks,
>>  >S.
>>  >
>>  >пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл < tkalkirill@yandex.ru
>:
>>  >
>>  >> Now, after restarting node, we have only cold caches, which at first
>>  >> requests to them will gradually load data from disks, which can slow
>>  down
>>  >> first calls to them.
>>  >> If node has more RAM than data on disk, then they can be loaded at start
>>  >> "warmup", thereby solving the issue of slowdowns during first calls to
>>  >> caches.
>>  >>
>>  >> I suggest adding a warmup phase after recovery here [1] after [2],
>>  before
>>  >> descovery.
>>  >>
>>  >> I suggest adding a new interface:
>>  >>
>>  >> package org.apache.ignite.internal.processors.cache;
>>  >>
>>  >> import org.apache.ignite.IgniteCheckedException;
>>  >> import org.apache.ignite.internal.IgniteInternalFuture;
>>  >> import org.jetbrains.annotations.Nullable;
>>  >>
>>  >> /**
>>  >> * Interface for warming up cache.
>>  >> */
>>  >> public interface CacheWarmup {
>>  >> /**
>>  >> * Warmup cache.
>>  >> *
>>  >> * @param cacheCtx Cache context.
>>  >> * @return Future cache warmup.
>>  >> * @throws IgniteCheckedException if failed.
>>  >> */
>>  >> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>  >> throws IgniteCheckedException;
>>  >> }
>>  >>
>>  >> Which will allow to warm up caches in parallel and asynchronously.
>>  Warmup
>>  >> phase will end after all IgniteInternalFuture for all caches isDone.
>>  >>
>>  >> Also adding the ability to customize via methods:
>>  >>
>>  org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>  >> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>  >>
>>  >> Which will allow for each cache to set implementation of cache warming
>>  >> up,
>>  >> both for a specific cache, and for all if necessary.
>>  >>
>>  >> I suggest adding an implementation of SequentialWarmup that will use
>>  [3].
>>  >>
>>  >> Questions, suggestions, comments?
>>  >>
>>  >> [1] -
>>  >>
>>  org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>  >> [2] -
>>  >>
>>  org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>  >> [3] -
>>  >>
>>  org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload

Mime
View raw message