ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ткаленко кирилл <tkalkir...@yandex.ru>
Subject Re: [DISCUSSION] Cache warmup
Date Mon, 10 Aug 2020 07:20:17 GMT
Hi, Stan!

As a result of personal correspondence I realized that you are right about making changes:
1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
2)Start warming up for each region sequentially;
3)Improving warm-up interface:

package org.apache.ignite.internal.processors.cache.warmup;

import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;

/**
 * Interface for warming up.
 */
public interface WarmUpStrategy<T extends WarmUpConfiguration> {
    /**
     * Returns configuration class for mapping to strategy.
     *
     * @return Configuration class.
     */
    Class<T> configClass();

    /**
     * Warm up.
     *
     * @param kernalCtx Kernal context.
     * @param cfg       Warm-up configuration.
     * @param region    Data region.
     * @throws IgniteCheckedException if faild.
     */
    void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;

    /**
     * Closing warm up.
     *
     * @throws IgniteCheckedException if faild.
     */
    void close() throws IgniteCheckedException;
}

4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up
stop
5)The "load all" strategy will work as long as there is enough RAM and index pages will also
take priority.

04.08.2020, 13:29, "ткаленко кирилл" <tkalkirill@yandex.ru>:
> Hi, Slava!
>
> Thank you for looking at the offer and making fair comments.
>
> I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40"
and we found out that point 2 in it is no longer relevant and it can be removed.
> I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up
will always start after restore phase, without subscribing to events.
>
> You are right this should be mentioned in the documentation and javadoc.
>>  This means that the user's thread, which starts Ignite via
>>  Ignition.start(), will wait for ana additional step - cache warm-up.
>>  I think this fact has to be clearly mentioned in our documentation (at
>>  Javadocat least) because this step can be time-consuming.
>
> My suggestion for implementation:
> 1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for
configuring cache warming;
> 2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
> 3)Add an internal warm-up interface that will start in [1] after [2];
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
>
> /**
>  * Interface for warming up.
>  */
> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>     /**
>      * Returns configuration class for mapping to strategy.
>      *
>      * @return Configuration class.
>      */
>     Class<T> configClass();
>
>     /**
>      * Warm up.
>      *
>      * @param kernalCtx Kernal context.
>      * @param cfg Warm-up configuration.
>      * @throws IgniteCheckedException if faild.
>      */
>     void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
> }
>
> 4)Adding an internal plugin extension for add own strategies;
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import java.util.Collection;
> import org.apache.ignite.plugin.Extension;
>
> /**
>  * Interface for getting warm-up strategies from plugins.
>  */
> public interface WarmUpStrategySupplier extends Extension {
>     /**
>      * Getting warm-up strategies.
>      *
>      * @return Warm-up strategies.
>      */
>     Collection<WarmUpStrategy> strategies();
> }
>
> 5)Add a "Load all" strategy that will load everything to memory as long as there is space
in it. This strategy is suitable if the persistent storage is less than RAM.
>
> Any objections or comments?
>
> [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
> [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>
> 27.07.2020, 16:48, "Вячеслав Коптилин" <slava.koptilin@gmail.com>:
>>  Hello Kirill,
>>
>>  Thanks a lot for driving this activity. If I am not mistaken, this
>>  discussion relates to IEP-40.
>>
>>>   I suggest adding a warmup phase after recovery here [1] after [2], before
>>
>>  discovery.
>>  This means that the user's thread, which starts Ignite via
>>  Ignition.start(), will wait for ana additional step - cache warm-up.
>>  I think this fact has to be clearly mentioned in our documentation (at
>>  Javadocat least) because this step can be time-consuming.
>>
>>>   I suggest adding a new interface:
>>
>>  I would change it a bit. First of all, it would be nice to place this
>>  interface to a public package and get rid of using GridCacheContext,
>>  which is an internal class and it should not leak to the public API in any
>>  case.
>>  Perhaps, this parameter is not needed at all or we should add some public
>>  abstraction instead of internal class.
>>
>>  package org.apache.ignite.configuration;
>>
>>  import org.apache.ignite.IgniteCheckedException;
>>  import org.apache.ignite.lang.IgniteFuture;
>>
>>  public interface CacheWarmupper {
>>      /**
>>       * Warmup cache.
>>       *
>>       * @param cachename Cache name.
>>       * @return Future cache warmup.
>>       * @throws IgniteCheckedException If failed.
>>       */
>>      IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>  }
>>
>>  Thanks,
>>  S.
>>
>>  пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tkalkirill@yandex.ru>:
>>
>>>   Now, after restarting node, we have only cold caches, which at first
>>>   requests to them will gradually load data from disks, which can slow down
>>>   first calls to them.
>>>   If node has more RAM than data on disk, then they can be loaded at start
>>>   "warmup", thereby solving the issue of slowdowns during first calls to
>>>   caches.
>>>
>>>   I suggest adding a warmup phase after recovery here [1] after [2], before
>>>   descovery.
>>>
>>>   I suggest adding a new interface:
>>>
>>>   package org.apache.ignite.internal.processors.cache;
>>>
>>>   import org.apache.ignite.IgniteCheckedException;
>>>   import org.apache.ignite.internal.IgniteInternalFuture;
>>>   import org.jetbrains.annotations.Nullable;
>>>
>>>   /**
>>>    * Interface for warming up cache.
>>>    */
>>>   public interface CacheWarmup {
>>>       /**
>>>        * Warmup cache.
>>>        *
>>>        * @param cacheCtx Cache context.
>>>        * @return Future cache warmup.
>>>        * @throws IgniteCheckedException if failed.
>>>        */
>>>       @Nullable IgniteInternalFuture<?> process(GridCacheContext
cacheCtx)
>>>   throws IgniteCheckedException;
>>>   }
>>>
>>>   Which will allow to warm up caches in parallel and asynchronously. Warmup
>>>   phase will end after all IgniteInternalFuture for all caches isDone.
>>>
>>>   Also adding the ability to customize via methods:
>>>   org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>   org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>
>>>   Which will allow for each cache to set implementation of cache warming up,
>>>   both for a specific cache, and for all if necessary.
>>>
>>>   I suggest adding an implementation of SequentialWarmup that will use [3].
>>>
>>>   Questions, suggestions, comments?
>>>
>>>   [1] -
>>>   org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>   [2] -
>>>   org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>   [3] -
>>>   org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload

Mime
View raw message