ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Goncharuk <alexey.goncha...@gmail.com>
Subject Re: Eviction policies with persistence
Date Wed, 07 Feb 2018 07:25:46 GMT
Denis,

Yes, if there are no concurrent updates and there are no ongoing
transactions preventing the entries to be evicted, then the whole page will
be purged.

2018-02-07 0:23 GMT+03:00 Denis Magda <dmagda@apache.org>:

> Alexey,
>
> Let me understand this part because it’s still not crystal clear to me:
>
> > Now as per the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
>
>
> However will all the keys-values from a chosen page be removed eventually
> during the eviction phase? If it’s true then it’s still a page based
> eviction - we select 5 random oldest pages and purge all the key-values
> from them.
>
> —
> Denis
>
> > On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
> >
> > Guys,
> >
> > To clarify the questions, I would like to reiterate over the mechanics of
> > evictions and then suggest a renaming that should resolve such things in
> > the future.
> >
> > First of all, Lucas is right - eviction policy only makes sense when
> native
> > persistence is disabled because what it does is actually freeing up
> memory
> > when a user hits the memory limit. The only way to do this is to destroy
> > inserted data because there is no other way to free memory. Now as per
> the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
> > If there are no concurrent updates, the page becomes empty and will be
> > reused for other user data. This is exactly what is explained on the wiki
> > page (at least in my reading of the page).
> >
> > Second, at this point, I would rename page management mechanism with
> > enabled persistence from 'eviction' to 'page replacement' or 'page
> > swapping', because it does not destroy any data, but rather replaces a
> > chosen buffer in memory from one page to another. The content of neither
> > pages does not change during page replacement. This mechanism is unlikely
> > to be selected by a user because the effectiveness of page replacements
> > heavily depends on internal data structures and may change from version
> to
> > version, and may even be adaptive depending on the load pattern.
> >
> > Hope this resolves the confusion.
> >
> > --AG
> >
> > 2018-02-03 1:03 GMT+03:00 Denis Magda <dmagda@apache.org>:
> >
> >> Dmitriy,
> >>
> >> I’m surprised to hear that Random-LRU works at the entry level when
> Ignite
> >> persistence is off. Frankly, this section confuses a lot:
> >> https://cwiki.apache.org/confluence/display/IGNITE/
> >> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >> underthehood-Entryeviction <https://cwiki.apache.org/
> >> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> >> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >>
> >> While it was assumed that the entry-based eviction works only for
> on-heap
> >> caches:
> >> https://apacheignite.readme.io/docs/evictions <
> >> https://apacheignite.readme.io/docs/evictions>
> >>
> >> Alex G., please chime in and clarify the misunderstanding.
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
> wrote:
> >>>
> >>> Hi Val,
> >>>
> >>> I think it is quite accurate because eviction in case PDS enabled or
> not
> >>> has quite different purposes.
> >>>
> >>> 1. Let us consider PDS enabled and page eviction occurs. First of all
> it
> >> is
> >>> page based eviction, but not entry-based. Actually data is not removed,
> >> but
> >>> only written to disk. We can address this page later by ID.
> >>> PDS eviction is primarily the replacement of pages from memory to page
> on
> >>> disk. This eviction policy should ensure the maximum performance of the
> >>> solution in the future. There is no data removal from grid in this
> case.
> >>> And Ignite does not allow users to configure the policy. If benchmarks
> >> show
> >>> that a change in policy results in increased performance, then we can
> >>> switch policy.
> >>>
> >>> 2. Entry-based eviction (if there is no persistence is available) is
> >>> slithly different. User will need to reload data into grid in case
> entry
> >> is
> >>> evicted but required in cache. In that case Ignite provides selection
> of
> >>> policies.
> >>>
> >>> Sincerely,
> >>> Dmirtiy Pavlov
> >>>
> >>>
> >>>
> >>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> >>> valentin.kulichenko@gmail.com>:
> >>>
> >>>> Guys,
> >>>>
> >>>> Thanks for responses. But I still fail to understand how it affects
a
> >> user.
> >>>>
> >>>> Are you saying that with persistence enabled Random-LRU is always used
> >>>> regardless of what is specified in pageEvictionMode, while in
> in-memory
> >>>> only scenario we can also utilize Random-2-LRU for eviction? Is this
> >>>> accurate? Are there any other differences?
> >>>>
> >>>> -Val
> >>>>
> >>>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Vyacheslav,
> >>>>>
> >>>>> Yes, this is right, but for now PDS page based eviction uses
> >> Random-LRU,
> >>>>> not Random-2-LRU. Something may be changed in future Ignite releases,
> >> but
> >>>>> now Random-LRU is used. I am going to research if there is some
room
> >> for
> >>>>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
> >>>>>
> >>>>> If clean page is evicted, nothing really happends. If dirty page
is
> >>>> evicted
> >>>>> from region during checkpointing process run, page is stored in
file
> >>>> before
> >>>>> eviction
> >>>>>
> >>>>> Some info about this can be found in
> >>>>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >>>>> underthehood-Pagebasedeviction
> >>>>>
> >>>>> Sincerely,
> >>>>> Dmitriy Pavlov
> >>>>>
> >>>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur <daradurvs@gmail.com
> >:
> >>>>>
> >>>>>> Hi Valentin,
> >>>>>>
> >>>>>> As far as I know, when persistence is enabled and Ignite can't
> >>>>>> allocate new page-memory in the memory segment, then Ignite
> >>>>>> automatically evict some data from RAM to PDS using Random-2-LRU
> >>>>>> eviction algorithm. And there are no mechanisms to evict entries
out
> >>>>>> when PDS is enabled.
> >>>>>>
> >>>>>> I'll be glad if I am not right and someone will correct me because,
> as
> >>>>>> Ignite user, I can't use PDS only as RAM disk replication for
the
> use
> >>>>>> case in my project.
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
> >>>>>> <valentin.kulichenko@gmail.com> wrote:
> >>>>>>> Folks,
> >>>>>>>
> >>>>>>> On "Eviction Policies" documentation page [1] we have the
following
> >>>>>> callout:
> >>>>>>>
> >>>>>>>> Configured eviction policy has no effect if Ignite persistence
is
> >>>>>> enabled
> >>>>>>>> Note that if Ignite Persistence is enabled, then the
page-based
> >>>>>> evictions
> >>>>>>> have no effect because the oldest pages will be purged from
memory
> >>>>>>> automatically.
> >>>>>>>
> >>>>>>> This really confuses me. Why is there a difference in how
data is
> >>>>> evicted
> >>>>>>> from memory depending on having disk enabled or not? And
how does
> >>>>>> eviction
> >>>>>>> work with persistence enabled then?
> >>>>>>>
> >>>>>>> Does anyone can clarify?
> >>>>>>>
> >>>>>>> [1] https://apacheignite.readme.io/docs/evictions
> >>>>>>>
> >>>>>>> -Val
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best Regards, Vyacheslav D.
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message