ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanislav Lukyanov <stanlukya...@gmail.com>
Subject Re: [DISCUSSION] Cache warmup
Date Mon, 10 Aug 2020 13:30:04 GMT
All of this looks awesome, covers the use cases I know about.
Thanks!

Stan

> On 10 Aug 2020, at 15:39, ткаленко кирилл <tkalkirill@yandex.ru>
wrote:
> 
> Hi, Stan again :-)
> 
> I suggest adding a little more flexibility to configuration:
> 1)Add default warm-up configuration for all regions into org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
> 2)Add a NoOp strategy for turning off heating of a specific region
> 
> Thus, when starting warm-up, region configuration is taken at beginning, and if it is
not present, it is taken from default. And if we don't want to warm up region, we set NoOp.
> 
> 10.08.2020, 10:20, "ткаленко кирилл" <tkalkirill@yandex.ru>:
>> Hi, Stan!
>> 
>> As a result of personal correspondence I realized that you are right about making
changes:
>> 1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>> 2)Start warming up for each region sequentially;
>> 3)Improving warm-up interface:
>> 
>> package org.apache.ignite.internal.processors.cache.warmup;
>> 
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.configuration.WarmUpConfiguration;
>> import org.apache.ignite.internal.GridKernalContext;
>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>> 
>> /**
>>  * Interface for warming up.
>>  */
>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>     /**
>>      * Returns configuration class for mapping to strategy.
>>      *
>>      * @return Configuration class.
>>      */
>>     Class<T> configClass();
>> 
>>     /**
>>      * Warm up.
>>      *
>>      * @param kernalCtx Kernal context.
>>      * @param cfg Warm-up configuration.
>>      * @param region Data region.
>>      * @throws IgniteCheckedException if faild.
>>      */
>>     void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
>> 
>>     /**
>>      * Closing warm up.
>>      *
>>      * @throws IgniteCheckedException if faild.
>>      */
>>     void close() throws IgniteCheckedException;
>> }
>> 
>> 4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up
stop
>> 5)The "load all" strategy will work as long as there is enough RAM and index pages
will also take priority.
>> 
>> 04.08.2020, 13:29, "ткаленко кирилл" <tkalkirill@yandex.ru>:
>>>  Hi, Slava!
>>> 
>>>  Thank you for looking at the offer and making fair comments.
>>> 
>>>  I personally discussed with Anton and Alexey because they are author and sponsor
of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
>>>  I suggest implementing point 3, since it may be independent of point 1. Also,
the warm-up will always start after restore phase, without subscribing to events.
>>> 
>>>  You are right this should be mentioned in the documentation and javadoc.
>>>>   This means that the user's thread, which starts Ignite via
>>>>   Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>   I think this fact has to be clearly mentioned in our documentation (at
>>>>   Javadocat least) because this step can be time-consuming.
>>> 
>>>  My suggestion for implementation:
>>>  1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration"
for configuring cache warming;
>>>  2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>>  3)Add an internal warm-up interface that will start in [1] after [2];
>>> 
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>> 
>>>  import org.apache.ignite.IgniteCheckedException;
>>>  import org.apache.ignite.configuration.WarmUpConfiguration;
>>>  import org.apache.ignite.internal.GridKernalContext;
>>> 
>>>  /**
>>>   * Interface for warming up.
>>>   */
>>>  public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>>      /**
>>>       * Returns configuration class for mapping to strategy.
>>>       *
>>>       * @return Configuration class.
>>>       */
>>>      Class<T> configClass();
>>> 
>>>      /**
>>>       * Warm up.
>>>       *
>>>       * @param kernalCtx Kernal context.
>>>       * @param cfg Warm-up configuration.
>>>       * @throws IgniteCheckedException if faild.
>>>       */
>>>      void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
>>>  }
>>> 
>>>  4)Adding an internal plugin extension for add own strategies;
>>> 
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>> 
>>>  import java.util.Collection;
>>>  import org.apache.ignite.plugin.Extension;
>>> 
>>>  /**
>>>   * Interface for getting warm-up strategies from plugins.
>>>   */
>>>  public interface WarmUpStrategySupplier extends Extension {
>>>      /**
>>>       * Getting warm-up strategies.
>>>       *
>>>       * @return Warm-up strategies.
>>>       */
>>>      Collection<WarmUpStrategy> strategies();
>>>  }
>>> 
>>>  5)Add a "Load all" strategy that will load everything to memory as long as there
is space in it. This strategy is suitable if the persistent storage is less than RAM.
>>> 
>>>  Any objections or comments?
>>> 
>>>  [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>  [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>> 
>>>  27.07.2020, 16:48, "Вячеслав Коптилин" <slava.koptilin@gmail.com>:
>>>>   Hello Kirill,
>>>> 
>>>>   Thanks a lot for driving this activity. If I am not mistaken, this
>>>>   discussion relates to IEP-40.
>>>> 
>>>>>    I suggest adding a warmup phase after recovery here [1] after [2],
before
>>>> 
>>>>   discovery.
>>>>   This means that the user's thread, which starts Ignite via
>>>>   Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>   I think this fact has to be clearly mentioned in our documentation (at
>>>>   Javadocat least) because this step can be time-consuming.
>>>> 
>>>>>    I suggest adding a new interface:
>>>> 
>>>>   I would change it a bit. First of all, it would be nice to place this
>>>>   interface to a public package and get rid of using GridCacheContext,
>>>>   which is an internal class and it should not leak to the public API in
any
>>>>   case.
>>>>   Perhaps, this parameter is not needed at all or we should add some public
>>>>   abstraction instead of internal class.
>>>> 
>>>>   package org.apache.ignite.configuration;
>>>> 
>>>>   import org.apache.ignite.IgniteCheckedException;
>>>>   import org.apache.ignite.lang.IgniteFuture;
>>>> 
>>>>   public interface CacheWarmupper {
>>>>       /**
>>>>        * Warmup cache.
>>>>        *
>>>>        * @param cachename Cache name.
>>>>        * @return Future cache warmup.
>>>>        * @throws IgniteCheckedException If failed.
>>>>        */
>>>>       IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>>>   }
>>>> 
>>>>   Thanks,
>>>>   S.
>>>> 
>>>>   пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tkalkirill@yandex.ru>:
>>>> 
>>>>>    Now, after restarting node, we have only cold caches, which at first
>>>>>    requests to them will gradually load data from disks, which can slow
down
>>>>>    first calls to them.
>>>>>    If node has more RAM than data on disk, then they can be loaded at
start
>>>>>    "warmup", thereby solving the issue of slowdowns during first calls
to
>>>>>    caches.
>>>>> 
>>>>>    I suggest adding a warmup phase after recovery here [1] after [2],
before
>>>>>    descovery.
>>>>> 
>>>>>    I suggest adding a new interface:
>>>>> 
>>>>>    package org.apache.ignite.internal.processors.cache;
>>>>> 
>>>>>    import org.apache.ignite.IgniteCheckedException;
>>>>>    import org.apache.ignite.internal.IgniteInternalFuture;
>>>>>    import org.jetbrains.annotations.Nullable;
>>>>> 
>>>>>    /**
>>>>>     * Interface for warming up cache.
>>>>>     */
>>>>>    public interface CacheWarmup {
>>>>>        /**
>>>>>         * Warmup cache.
>>>>>         *
>>>>>         * @param cacheCtx Cache context.
>>>>>         * @return Future cache warmup.
>>>>>         * @throws IgniteCheckedException if failed.
>>>>>         */
>>>>>        @Nullable IgniteInternalFuture<?> process(GridCacheContext
cacheCtx)
>>>>>    throws IgniteCheckedException;
>>>>>    }
>>>>> 
>>>>>    Which will allow to warm up caches in parallel and asynchronously.
Warmup
>>>>>    phase will end after all IgniteInternalFuture for all caches isDone.
>>>>> 
>>>>>    Also adding the ability to customize via methods:
>>>>>    org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>>    org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>> 
>>>>>    Which will allow for each cache to set implementation of cache warming
up,
>>>>>    both for a specific cache, and for all if necessary.
>>>>> 
>>>>>    I suggest adding an implementation of SequentialWarmup that will use
[3].
>>>>> 
>>>>>    Questions, suggestions, comments?
>>>>> 
>>>>>    [1] -
>>>>>    org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>>    [2] -
>>>>>    org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>>    [3] -
>>>>>    org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload


Mime
View raw message