ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Persistence per memory policy configuration
Date Fri, 13 Oct 2017 02:01:34 GMT
>From what I see after running an example they are under the same root
folder and in different subdirectories. The root folder should be defined
by setPersistencePath as I guess.

Denis

On Thursday, October 12, 2017, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> On Thu, Oct 12, 2017 at 4:05 PM, Denis Magda <dmagda@apache.org
> <javascript:;>> wrote:
>
> > Both terms “persistence” and “storage” mean same to me. If to choose
> > between them I would go for “persistence” because this is how
> > we name the feature - “Ignite Persistence” [1]
>
>
> > Furthermore, by default WAL files, the archive and data/index files are
> > located under the same root which is “setPersistencePath” directory.
> >
>
> Denis, to my knowledge, WAL can span multiple caches, no? How can it end up
> in the same folder with data and indexes, if those are per cache?
>
>
> >
> > [1] https://ignite.apache.org/features/persistence.html <
> > https://ignite.apache.org/features/persistence.html>
> >
> > —
> > Denis
> >
> > > On Oct 12, 2017, at 12:00 AM, Ivan Rakov <ivan.glukos@gmail.com
> <javascript:;>> wrote:
> > >
> > > Name *setPersistencePath* looks confusing to me because both WAL and
> > index/partition files storage provide persistence. That's why we
> separated
> > API methods as *setWalPath* and *setStoragePath*.
> > > I think, *setStoragePath* is good enough as long as it's supported by
> > explaining javadoc.*
> > > *
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > > On 12.10.2017 8:05, Dmitriy Setrakyan wrote:
> > >> Is the storage path the root folder for the persistence or only the
> root
> > >> path for the main storage?
> > >>
> > >> On Wed, Oct 11, 2017 at 3:54 PM, Denis Magda <dmagda@apache.org
> <javascript:;>> wrote:
> > >>
> > >>> Ivan,
> > >>>
> > >>> Instead of “setStoragePath” I would suggest “setPersistencePath”.
> Left
> > >>> some extra notes in the ticket.
> > >>>
> > >>
> > >>> —
> > >>> Denis
> > >>>
> > >>>> On Oct 11, 2017, at 4:30 AM, Ivan Rakov <ivan.glukos@gmail.com
> <javascript:;>>
> > wrote:
> > >>>>
> > >>>> Vladimir,
> > >>>>
> > >>>> Thanks for focusing on existing namings. Most of your suggestions
> > really
> > >>> sound better. I've posted my thoughts under your comment.
> > >>>> By the way, we should decide two things:
> > >>>>
> > >>>> 1) Naming for methods for configuring store path. I suggest the
> > >>> following:
> > >>>> *setStoragePath* - for partition and index files
> > >>>> *setWalPath* - for WAL files
> > >>>> *walArchivePath* - for WAL archive files
> > >>>>
> > >>>> 2) Renaming *checkpointingFrequency* to *checkpointFrequency* (same
> > with
> > >>> *checkpointingPageBufferSize* and *checkpointingThreads*). Both
> options
> > >>> sounds ok to me, let's see what community thinks.
> > >>>> Best Regards,
> > >>>> Ivan Rakov
> > >>>>
> > >>>> On 11.10.2017 14:05, Vladimir Ozerov wrote:
> > >>>>> Ivan,
> > >>>>>
> > >>>>> I left some comments in the ticket [1], please take a look.
> > >>>>>
> > >>>>> [1]
> > >>>>> https://issues.apache.org/jira/browse/IGNITE-6030?
> > >>> focusedCommentId=16200050&page=com.atlassian.jira.
> > >>> plugin.system.issuetabpanels:comment-tabpanel#comment-16200050
> > >>>>> On Wed, Oct 11, 2017 at 12:04 PM, Ivan Rakov <
> ivan.glukos@gmail.com <javascript:;>>
> > >>> wrote:
> > >>>>>> Igniters,
> > >>>>>>
> > >>>>>> https://issues.apache.org/jira/browse/IGNITE-6030 is ready
and
> > >>> enqueued
> > >>>>>> for TC run.
> > >>>>>> PR: https://github.com/apache/ignite/pull/2828
> > >>>>>>
> > >>>>>> Everyone interested in new data storage configuration API,
please
> > pay
> > >>>>>> attention and review.
> > >>>>>>
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Ivan Rakov
> > >>>>>>
> > >>>>>>
> > >>>>>> On 09.10.2017 12:40, Pavel Tupitsyn wrote:
> > >>>>>>
> > >>>>>>> Sounds good to me.
> > >>>>>>>
> > >>>>>>> On Mon, Oct 9, 2017 at 12:35 PM, Ivan Rakov <
> ivan.glukos@gmail.com <javascript:;>
> > >
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Pavel,
> > >>>>>>>> Sounds reasonable.
> > >>>>>>>> I suggest to include both "data" and "configuration"
to make it
> > even
> > >>> more
> > >>>>>>>> obvious:
> > >>>>>>>>
> > >>>>>>>> set/getDefaultDataRegionConfiguration
> > >>>>>>>> set/getDataRegionConfigurations
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Ivan Rakov
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 09.10.2017 10:51, Pavel Tupitsyn wrote:
> > >>>>>>>>
> > >>>>>>>> Sorry that I'm late to the party, but this looks
inconsistent:
> > >>>>>>>>> DataStorageConfiguration defaultRegionConfiguration
> > >>>>>>>>> DataRegionConfiguration[] getDataRegions
> > >>>>>>>>>
> > >>>>>>>>> defaultRegionConfiguration + getRegionConfigurations
> > >>>>>>>>> - or -
> > >>>>>>>>> defaultDataRegion + getDataRegions
> > >>>>>>>>>
> > >>>>>>>>> Thoughts?
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Oct 2, 2017 at 9:10 PM, Ivan Rakov
<
> > ivan.glukos@gmail.com <javascript:;>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Denis,
> > >>>>>>>>>
> > >>>>>>>>>> Yes, you're right. All cache groups without
specific data
> region
> > >>>>>>>>>> configured will be persistent.
> > >>>>>>>>>> And if you want to add another persistent
data region, you
> > should
> > >>> set
> > >>>>>>>>>> *isPeristenceEnabled* flag in its *DataRegionConfiguration*
> > >>> explictly.
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Ivan Rakov
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 02.10.2017 21:01, Denis Magda wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Missed the point with defaults. Makes sense
to me now. So to
> > wrap
> > >>> this
> > >>>>>>>>>>> up, if I want to enable the persistence
globally and don’t
> have
> > >>> any
> > >>>>>>>>>>> regions
> > >>>>>>>>>>> configured explicitly I need to take
the default region and
> > >>> switch the
> > >>>>>>>>>>> persistence on for it. Is my understanding
correct?
> > >>>>>>>>>>>
> > >>>>>>>>>>> —
> > >>>>>>>>>>> Denis
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Oct 2, 2017, at 10:57 AM, Ivan Rakov
<
> ivan.glukos@gmail.com <javascript:;>
> > >
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Denis, why do you need to access an
instance of the default
> > region
> > >>>>>>>>>>>> bean?
> > >>>>>>>>>>>> If you want to set any parameter,
just instantiate new bean
> > with
> > >>> this
> > >>>>>>>>>>>> parameter set (like in XML snipped
below). Other parameters
> > will
> > >>> be
> > >>>>>>>>>>>> automatically initialized with
their default values.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Ivan Rakov
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 02.10.2017 19:28, Denis Magda
wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>         <property name="dataStorageConfiguration">
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>             <bean class="org.apache.ignite.confi
> > >>>>>>>>>>>>>>> guration.DataStorageConfiguration">
> > >>>>>>>>>>>>>>>                 <property
name="systemCacheInitialSize"
> > >>>>>>>>>>>>>>> value="#{100
> > >>>>>>>>>>>>>>> *
> > >>>>>>>>>>>>>>> 1024 * 1024}"/>
> > >>>>>>>>>>>>>>>                 <property
name="
> > defaultRegionConfiguration">
> > >>>>>>>>>>>>>>>                   
 <bean class="org.apache.ignite.confi
> > >>>>>>>>>>>>>>> guration.DataRegionConfiguration">
> > >>>>>>>>>>>>>>>                   
     <property name="maxSize"
> > value="#{5 *
> > >>>>>>>>>>>>>>> 1024 *
> > >>>>>>>>>>>>>>> 102 * 1024}"/>
> > >>>>>>>>>>>>>>>                   
 </bean>
> > >>>>>>>>>>>>>>>                 </property>
> > >>>>>>>>>>>>>>>             </bean>
> > >>>>>>>>>>>>>>>         </property>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In other data regions
persistence will be disabled by
> > default.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ivan, how to get an instance
to the default region bean
> and
> > >>> change
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> parameter? Obviously, if the
goal is to enable the
> > persistence I
> > >>>>>>>>>>>>> don’t want
> > >>>>>>>>>>>>> to create the default region
bean from scratch.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> —
> > >>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Oct 2, 2017, at 9:11 AM,
Ivan Rakov <
> > ivan.glukos@gmail.com <javascript:;>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Agree with Alexey.
> > >>>>>>>>>>>>>> Properties like *defaultDataRegionSize*,
> > >>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> > >>>>>>>>>>>>>> can confuse users who don't
know that there's such thing
> as
> > >>> default
> > >>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>> region. They can decide
they are inherited by all data
> > regions
> > >>>>>>>>>>>>>> where
> > >>>>>>>>>>>>>> size
> > >>>>>>>>>>>>>> and persistence flag are
not explicitly set.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let's get rid of these
properties and add
> > >>>>>>>>>>>>>> *defaultRegionConfiguration*
> > >>>>>>>>>>>>>> property with explicit
configuration of default data
> region.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Regarding XML configuration,
changing size or persistence
> > flag
> > >>> of
> > >>>>>>>>>>>>>> default data region will
be just two lines longer (for
> bean
> > >>>>>>>>>>>>>> description):
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>         <property name="dataStorageConfiguration">
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>             <bean class="org.apache.ignite.confi
> > >>>>>>>>>>>>>>> guration.DataStorageConfiguration">
> > >>>>>>>>>>>>>>>                 <property
name="systemCacheInitialSize"
> > >>>>>>>>>>>>>>> value="#{100
> > >>>>>>>>>>>>>>> *
> > >>>>>>>>>>>>>>> 1024 * 1024}"/>
> > >>>>>>>>>>>>>>>                 <property
name="
> > defaultRegionConfiguration">
> > >>>>>>>>>>>>>>>                   
 <bean class="org.apache.ignite.confi
> > >>>>>>>>>>>>>>> guration.DataRegionConfiguration">
> > >>>>>>>>>>>>>>>                   
     <property name="maxSize"
> > value="#{5 *
> > >>>>>>>>>>>>>>> 1024 *
> > >>>>>>>>>>>>>>> 102 * 1024}"/>
> > >>>>>>>>>>>>>>>                   
 </bean>
> > >>>>>>>>>>>>>>>                 </property>
> > >>>>>>>>>>>>>>>             </bean>
> > >>>>>>>>>>>>>>>         </property>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In other data regions
persistence will be disabled by
> > default.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I've updated draft in https://issues.apache.org/jira
> > >>>>>>>>>>>>>> /browse/IGNITE-6030 with
these changes.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Ivan Rakov
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 02.10.2017 18:35, Denis
Magda wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To resolve this, I suggest
to
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> introduce just another
field defaultRegionConfiguration
> and
> > >>> get
> > >>>>>>>>>>>>>>>> rid
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> other defaults
in DataStorageConfiguration.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Won’t it complicate
the configuration from a Spring XML
> > >>> file? I’m
> > >>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> an expert in Spring
so how do I get
> > defaultRegionConfiguration
> > >>>>>>>>>>>>>>> bean
> > >>>>>>>>>>>>>>> first
> > >>>>>>>>>>>>>>> to change any parameter?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> —
> > >>>>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Oct 2, 2017, at
8:30 AM, Alexey Goncharuk <
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com
<javascript:;>> wrote:
> > >>>>>>>>>>>>>>>> Agree with Vladimir.
If we are to implement this, we
> would
> > >>> either
> > >>>>>>>>>>>>>>>> need to
> > >>>>>>>>>>>>>>>> have a Boolean
(non-primitive) for persistenceEnabled on
> > >>>>>>>>>>>>>>>> DataRegionConfiguration,
or introduce an enum for this
> > field
> > >>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>> is also
> > >>>>>>>>>>>>>>>> an overkill. On
the other hand, one can assume that the
> > >>> defaults
> > >>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>> talking about are
actually inherited. To resolve this, I
> > >>> suggest
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> introduce just
another field defaultRegionConfiguration
> > and
> > >>> get
> > >>>>>>>>>>>>>>>> rid
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> other defaults
in DataStorageConfiguration.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thoughts?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 2017-10-02 15:19
GMT+03:00 Ivan Rakov <
> > ivan.glukos@gmail.com <javascript:;>
> > >>>> :
> > >>>>>>>>>>>>>>>> Vladimir,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I like your approach
because it's easier to implement.
> > >>>>>>>>>>>>>>>>> However, user
may be confused by setting
> > >>>>>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> > >>>>>>>>>>>>>>>>> flag and seeing
that persistence is not enabled by
> > default
> > >>> in
> > >>>>>>>>>>>>>>>>> custom memory
> > >>>>>>>>>>>>>>>>> region. I'll
add clarifying Javadoc at this place.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Ivan Rakov
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 02.10.2017
11:28, Vladimir Ozerov wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Ivan,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I do not think
this is correct approach, because it
> will
> > be
> > >>> hard
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> explain,
and you will have to use "Boolean" instead of
> > >>>>>>>>>>>>>>>>>> "boolean"
> > >>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> DataRegionConfiguration.
I do not think we need
> default
> > >>>>>>>>>>>>>>>>>> "persistence
> > >>>>>>>>>>>>>>>>>> enabled"
for all regions. Instead, we should have
> > >>> "persistence
> > >>>>>>>>>>>>>>>>>> enabled"
> > >>>>>>>>>>>>>>>>>> flag for
default region only. It should not be
> > propagated
> > >>> to
> > >>>>>>>>>>>>>>>>>> custom
> > >>>>>>>>>>>>>>>>>> regions.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Mon,
Oct 2, 2017 at 11:14 AM, Ivan Rakov <
> > >>>>>>>>>>>>>>>>>> ivan.glukos@gmail.com
<javascript:;>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Guys, I
think I got the point now.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Let's check
the final design:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> *DataStorageConfiguration*
will have
> > >>>>>>>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> > >>>>>>>>>>>>>>>>>>> property
(default = false), which will be used for
> > >>> enabling
> > >>>>>>>>>>>>>>>>>>> persistence
> > >>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>> default
data region.
> > >>>>>>>>>>>>>>>>>>> *DataRegionConfiguration*
will have
> > *isPersistenceEnabled*
> > >>>>>>>>>>>>>>>>>>> property,
> > >>>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>> will
be used for enabling persistence in
> corresponding
> > >>> data
> > >>>>>>>>>>>>>>>>>>> region.
If
> > >>>>>>>>>>>>>>>>>>> value
is not set, value of
> > *DataStorageConfiguration::isD
> > >>>>>>>>>>>>>>>>>>> efaultPersistenceEnabled*
> > >>>>>>>>>>>>>>>>>>> will
be used by default.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best
Regards,
> > >>>>>>>>>>>>>>>>>>> Ivan
Rakov
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On
02.10.2017 7:49, Dmitriy Setrakyan wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On
Mon, Oct 2, 2017 at 7:46 AM, Denis Magda <
> > >>>>>>>>>>>>>>>>>>> dmagda@apache.org
<javascript:;>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On
Oct 1, 2017, at 4:41 AM, Ivan Rakov <
> > >>> ivan.glukos@gmail.com <javascript:;>
> > >>>>>>>>>>>>>>>>>>>>
wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
1) You're right. I forgot to include the main flag
> in
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
DataRegionConfiguration - *isPersistenceEnabled*.
> > >>> Persistence
> > >>>>>>>>>>>>>>>>>>>>>>
will be
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
enabled globally if at least one memory region has
> > this
> > >>>>>>>>>>>>>>>>>>>>>>
flag
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
set.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
I’m confused. Why the persistence should be enabled
> > >>>>>>>>>>>>>>>>>>>>>
*globally*
> > >>>>>>>>>>>>>>>>>>>>>
if the
> > >>>>>>>>>>>>>>>>>>>>>
purpose is to have it set for a specific region? If
> > it’s
> > >>>>>>>>>>>>>>>>>>>>>
enabled for
> > >>>>>>>>>>>>>>>>>>>>>
region
> > >>>>>>>>>>>>>>>>>>>>>
A only, I don’t want to have it activated for
> region
> > B.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
Yes, you are right. By default the persistence will
> > be
> > >>>>>>>>>>>>>>>>>>>>>
disabled
> > >>>>>>>>>>>>>>>>>>>>>
globally.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
But we should also give users a way to switch the
> > >>> default
> > >>>>>>>>>>>>>>>>>>>>>
behavior from
> > >>>>>>>>>>>>>>>>>>>>
in-memory only (no-persistence) to persistence.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message