ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ткаленко кирилл <tkalkir...@yandex.ru>
Subject Re: [DISCUSSION] Cache warmup
Date Mon, 10 Aug 2020 14:58:34 GMT
Great! Then I proceed to ticket https://issues.apache.org/jira/browse/IGNITE-13345.

10.08.2020, 16:30, "Stanislav Lukyanov" <stanlukyanov@gmail.com>:
> All of this looks awesome, covers the use cases I know about.
> Thanks!
>
> Stan
>
>>  On 10 Aug 2020, at 15:39, ткаленко кирилл <tkalkirill@yandex.ru>
wrote:
>>
>>  Hi, Stan again :-)
>>
>>  I suggest adding a little more flexibility to configuration:
>>  1)Add default warm-up configuration for all regions into org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
>>  2)Add a NoOp strategy for turning off heating of a specific region
>>
>>  Thus, when starting warm-up, region configuration is taken at beginning, and if
it is not present, it is taken from default. And if we don't want to warm up region, we set
NoOp.
>>
>>  10.08.2020, 10:20, "ткаленко кирилл" <tkalkirill@yandex.ru>:
>>>  Hi, Stan!
>>>
>>>  As a result of personal correspondence I realized that you are right about
making changes:
>>>  1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>>>  2)Start warming up for each region sequentially;
>>>  3)Improving warm-up interface:
>>>
>>>  package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>>  import org.apache.ignite.IgniteCheckedException;
>>>  import org.apache.ignite.configuration.WarmUpConfiguration;
>>>  import org.apache.ignite.internal.GridKernalContext;
>>>  import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>>
>>>  /**
>>>   * Interface for warming up.
>>>   */
>>>  public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>>      /**
>>>       * Returns configuration class for mapping to strategy.
>>>       *
>>>       * @return Configuration class.
>>>       */
>>>      Class<T> configClass();
>>>
>>>      /**
>>>       * Warm up.
>>>       *
>>>       * @param kernalCtx Kernal context.
>>>       * @param cfg Warm-up configuration.
>>>       * @param region Data region.
>>>       * @throws IgniteCheckedException if faild.
>>>       */
>>>      void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region)
throws IgniteCheckedException;
>>>
>>>      /**
>>>       * Closing warm up.
>>>       *
>>>       * @throws IgniteCheckedException if faild.
>>>       */
>>>      void close() throws IgniteCheckedException;
>>>  }
>>>
>>>  4)Add a command to "control.sh", to stop current warm-up and cancel all others:
--warm-up stop
>>>  5)The "load all" strategy will work as long as there is enough RAM and index
pages will also take priority.
>>>
>>>  04.08.2020, 13:29, "ткаленко кирилл" <tkalkirill@yandex.ru>:
>>>>   Hi, Slava!
>>>>
>>>>   Thank you for looking at the offer and making fair comments.
>>>>
>>>>   I personally discussed with Anton and Alexey because they are author
and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can
be removed.
>>>>   I suggest implementing point 3, since it may be independent of point
1. Also, the warm-up will always start after restore phase, without subscribing to events.
>>>>
>>>>   You are right this should be mentioned in the documentation and javadoc.
>>>>>    This means that the user's thread, which starts Ignite via
>>>>>    Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>>    I think this fact has to be clearly mentioned in our documentation
(at
>>>>>    Javadocat least) because this step can be time-consuming.
>>>>
>>>>   My suggestion for implementation:
>>>>   1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration"
for configuring cache warming;
>>>>   2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>>>   3)Add an internal warm-up interface that will start in [1] after [2];
>>>>
>>>>   package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>>   import org.apache.ignite.IgniteCheckedException;
>>>>   import org.apache.ignite.configuration.WarmUpConfiguration;
>>>>   import org.apache.ignite.internal.GridKernalContext;
>>>>
>>>>   /**
>>>>    * Interface for warming up.
>>>>    */
>>>>   public interface WarmUpStrategy<T extends WarmUpConfiguration>
{
>>>>       /**
>>>>        * Returns configuration class for mapping to strategy.
>>>>        *
>>>>        * @return Configuration class.
>>>>        */
>>>>       Class<T> configClass();
>>>>
>>>>       /**
>>>>        * Warm up.
>>>>        *
>>>>        * @param kernalCtx Kernal context.
>>>>        * @param cfg Warm-up configuration.
>>>>        * @throws IgniteCheckedException if faild.
>>>>        */
>>>>       void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
>>>>   }
>>>>
>>>>   4)Adding an internal plugin extension for add own strategies;
>>>>
>>>>   package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>>   import java.util.Collection;
>>>>   import org.apache.ignite.plugin.Extension;
>>>>
>>>>   /**
>>>>    * Interface for getting warm-up strategies from plugins.
>>>>    */
>>>>   public interface WarmUpStrategySupplier extends Extension {
>>>>       /**
>>>>        * Getting warm-up strategies.
>>>>        *
>>>>        * @return Warm-up strategies.
>>>>        */
>>>>       Collection<WarmUpStrategy> strategies();
>>>>   }
>>>>
>>>>   5)Add a "Load all" strategy that will load everything to memory as long
as there is space in it. This strategy is suitable if the persistent storage is less than
RAM.
>>>>
>>>>   Any objections or comments?
>>>>
>>>>   [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>   [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>
>>>>   27.07.2020, 16:48, "Вячеслав Коптилин" <slava.koptilin@gmail.com>:
>>>>>    Hello Kirill,
>>>>>
>>>>>    Thanks a lot for driving this activity. If I am not mistaken, this
>>>>>    discussion relates to IEP-40.
>>>>>
>>>>>>     I suggest adding a warmup phase after recovery here [1] after
[2], before
>>>>>
>>>>>    discovery.
>>>>>    This means that the user's thread, which starts Ignite via
>>>>>    Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>>    I think this fact has to be clearly mentioned in our documentation
(at
>>>>>    Javadocat least) because this step can be time-consuming.
>>>>>
>>>>>>     I suggest adding a new interface:
>>>>>
>>>>>    I would change it a bit. First of all, it would be nice to place
this
>>>>>    interface to a public package and get rid of using GridCacheContext,
>>>>>    which is an internal class and it should not leak to the public
API in any
>>>>>    case.
>>>>>    Perhaps, this parameter is not needed at all or we should add some
public
>>>>>    abstraction instead of internal class.
>>>>>
>>>>>    package org.apache.ignite.configuration;
>>>>>
>>>>>    import org.apache.ignite.IgniteCheckedException;
>>>>>    import org.apache.ignite.lang.IgniteFuture;
>>>>>
>>>>>    public interface CacheWarmupper {
>>>>>        /**
>>>>>         * Warmup cache.
>>>>>         *
>>>>>         * @param cachename Cache name.
>>>>>         * @return Future cache warmup.
>>>>>         * @throws IgniteCheckedException If failed.
>>>>>         */
>>>>>        IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>>>>    }
>>>>>
>>>>>    Thanks,
>>>>>    S.
>>>>>
>>>>>    пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл
<tkalkirill@yandex.ru>:
>>>>>
>>>>>>     Now, after restarting node, we have only cold caches, which
at first
>>>>>>     requests to them will gradually load data from disks, which
can slow down
>>>>>>     first calls to them.
>>>>>>     If node has more RAM than data on disk, then they can be
loaded at start
>>>>>>     "warmup", thereby solving the issue of slowdowns during first
calls to
>>>>>>     caches.
>>>>>>
>>>>>>     I suggest adding a warmup phase after recovery here [1] after
[2], before
>>>>>>     descovery.
>>>>>>
>>>>>>     I suggest adding a new interface:
>>>>>>
>>>>>>     package org.apache.ignite.internal.processors.cache;
>>>>>>
>>>>>>     import org.apache.ignite.IgniteCheckedException;
>>>>>>     import org.apache.ignite.internal.IgniteInternalFuture;
>>>>>>     import org.jetbrains.annotations.Nullable;
>>>>>>
>>>>>>     /**
>>>>>>      * Interface for warming up cache.
>>>>>>      */
>>>>>>     public interface CacheWarmup {
>>>>>>         /**
>>>>>>          * Warmup cache.
>>>>>>          *
>>>>>>          * @param cacheCtx Cache context.
>>>>>>          * @return Future cache warmup.
>>>>>>          * @throws IgniteCheckedException if failed.
>>>>>>          */
>>>>>>         @Nullable IgniteInternalFuture<?> process(GridCacheContext
cacheCtx)
>>>>>>     throws IgniteCheckedException;
>>>>>>     }
>>>>>>
>>>>>>     Which will allow to warm up caches in parallel and asynchronously.
Warmup
>>>>>>     phase will end after all IgniteInternalFuture for all caches
isDone.
>>>>>>
>>>>>>     Also adding the ability to customize via methods:
>>>>>>     org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>>>     org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>>>
>>>>>>     Which will allow for each cache to set implementation of
cache warming up,
>>>>>>     both for a specific cache, and for all if necessary.
>>>>>>
>>>>>>     I suggest adding an implementation of SequentialWarmup that
will use [3].
>>>>>>
>>>>>>     Questions, suggestions, comments?
>>>>>>
>>>>>>     [1] -
>>>>>>     org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>>>     [2] -
>>>>>>     org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>>>     [3] -
>>>>>>     org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload

Mime
View raw message