ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Persistence per memory policy configuration
Date Fri, 13 Oct 2017 01:26:35 GMT
On Thu, Oct 12, 2017 at 4:05 PM, Denis Magda <dmagda@apache.org> wrote:

> Both terms “persistence” and “storage” mean same to me. If to choose
> between them I would go for “persistence” because this is how
> we name the feature - “Ignite Persistence” [1]


> Furthermore, by default WAL files, the archive and data/index files are
> located under the same root which is “setPersistencePath” directory.
>

Denis, to my knowledge, WAL can span multiple caches, no? How can it end up
in the same folder with data and indexes, if those are per cache?


>
> [1] https://ignite.apache.org/features/persistence.html <
> https://ignite.apache.org/features/persistence.html>
>
> —
> Denis
>
> > On Oct 12, 2017, at 12:00 AM, Ivan Rakov <ivan.glukos@gmail.com> wrote:
> >
> > Name *setPersistencePath* looks confusing to me because both WAL and
> index/partition files storage provide persistence. That's why we separated
> API methods as *setWalPath* and *setStoragePath*.
> > I think, *setStoragePath* is good enough as long as it's supported by
> explaining javadoc.*
> > *
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 12.10.2017 8:05, Dmitriy Setrakyan wrote:
> >> Is the storage path the root folder for the persistence or only the root
> >> path for the main storage?
> >>
> >> On Wed, Oct 11, 2017 at 3:54 PM, Denis Magda <dmagda@apache.org> wrote:
> >>
> >>> Ivan,
> >>>
> >>> Instead of “setStoragePath” I would suggest “setPersistencePath”.
Left
> >>> some extra notes in the ticket.
> >>>
> >>
> >>> —
> >>> Denis
> >>>
> >>>> On Oct 11, 2017, at 4:30 AM, Ivan Rakov <ivan.glukos@gmail.com>
> wrote:
> >>>>
> >>>> Vladimir,
> >>>>
> >>>> Thanks for focusing on existing namings. Most of your suggestions
> really
> >>> sound better. I've posted my thoughts under your comment.
> >>>> By the way, we should decide two things:
> >>>>
> >>>> 1) Naming for methods for configuring store path. I suggest the
> >>> following:
> >>>> *setStoragePath* - for partition and index files
> >>>> *setWalPath* - for WAL files
> >>>> *walArchivePath* - for WAL archive files
> >>>>
> >>>> 2) Renaming *checkpointingFrequency* to *checkpointFrequency* (same
> with
> >>> *checkpointingPageBufferSize* and *checkpointingThreads*). Both options
> >>> sounds ok to me, let's see what community thinks.
> >>>> Best Regards,
> >>>> Ivan Rakov
> >>>>
> >>>> On 11.10.2017 14:05, Vladimir Ozerov wrote:
> >>>>> Ivan,
> >>>>>
> >>>>> I left some comments in the ticket [1], please take a look.
> >>>>>
> >>>>> [1]
> >>>>> https://issues.apache.org/jira/browse/IGNITE-6030?
> >>> focusedCommentId=16200050&page=com.atlassian.jira.
> >>> plugin.system.issuetabpanels:comment-tabpanel#comment-16200050
> >>>>> On Wed, Oct 11, 2017 at 12:04 PM, Ivan Rakov <ivan.glukos@gmail.com>
> >>> wrote:
> >>>>>> Igniters,
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/IGNITE-6030 is ready and
> >>> enqueued
> >>>>>> for TC run.
> >>>>>> PR: https://github.com/apache/ignite/pull/2828
> >>>>>>
> >>>>>> Everyone interested in new data storage configuration API, please
> pay
> >>>>>> attention and review.
> >>>>>>
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Ivan Rakov
> >>>>>>
> >>>>>>
> >>>>>> On 09.10.2017 12:40, Pavel Tupitsyn wrote:
> >>>>>>
> >>>>>>> Sounds good to me.
> >>>>>>>
> >>>>>>> On Mon, Oct 9, 2017 at 12:35 PM, Ivan Rakov <ivan.glukos@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Pavel,
> >>>>>>>> Sounds reasonable.
> >>>>>>>> I suggest to include both "data" and "configuration"
to make it
> even
> >>> more
> >>>>>>>> obvious:
> >>>>>>>>
> >>>>>>>> set/getDefaultDataRegionConfiguration
> >>>>>>>> set/getDataRegionConfigurations
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Ivan Rakov
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 09.10.2017 10:51, Pavel Tupitsyn wrote:
> >>>>>>>>
> >>>>>>>> Sorry that I'm late to the party, but this looks inconsistent:
> >>>>>>>>> DataStorageConfiguration defaultRegionConfiguration
> >>>>>>>>> DataRegionConfiguration[] getDataRegions
> >>>>>>>>>
> >>>>>>>>> defaultRegionConfiguration + getRegionConfigurations
> >>>>>>>>> - or -
> >>>>>>>>> defaultDataRegion + getDataRegions
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 2, 2017 at 9:10 PM, Ivan Rakov <
> ivan.glukos@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Denis,
> >>>>>>>>>
> >>>>>>>>>> Yes, you're right. All cache groups without
specific data region
> >>>>>>>>>> configured will be persistent.
> >>>>>>>>>> And if you want to add another persistent data
region, you
> should
> >>> set
> >>>>>>>>>> *isPeristenceEnabled* flag in its *DataRegionConfiguration*
> >>> explictly.
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Ivan Rakov
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 02.10.2017 21:01, Denis Magda wrote:
> >>>>>>>>>>
> >>>>>>>>>> Missed the point with defaults. Makes sense
to me now. So to
> wrap
> >>> this
> >>>>>>>>>>> up, if I want to enable the persistence
globally and don’t have
> >>> any
> >>>>>>>>>>> regions
> >>>>>>>>>>> configured explicitly I need to take the
default region and
> >>> switch the
> >>>>>>>>>>> persistence on for it. Is my understanding
correct?
> >>>>>>>>>>>
> >>>>>>>>>>> —
> >>>>>>>>>>> Denis
> >>>>>>>>>>>
> >>>>>>>>>>> On Oct 2, 2017, at 10:57 AM, Ivan Rakov
<ivan.glukos@gmail.com
> >
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Denis, why do you need to access an instance
of the default
> region
> >>>>>>>>>>>> bean?
> >>>>>>>>>>>> If you want to set any parameter, just
instantiate new bean
> with
> >>> this
> >>>>>>>>>>>> parameter set (like in XML snipped below).
Other parameters
> will
> >>> be
> >>>>>>>>>>>> automatically initialized with their
default values.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Ivan Rakov
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 02.10.2017 19:28, Denis Magda wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>         <property name="dataStorageConfiguration">
> >>>>>>>>>>>>
> >>>>>>>>>>>>>             <bean class="org.apache.ignite.confi
> >>>>>>>>>>>>>>> guration.DataStorageConfiguration">
> >>>>>>>>>>>>>>>                 <property
name="systemCacheInitialSize"
> >>>>>>>>>>>>>>> value="#{100
> >>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>> 1024 * 1024}"/>
> >>>>>>>>>>>>>>>                 <property
name="
> defaultRegionConfiguration">
> >>>>>>>>>>>>>>>                     <bean
class="org.apache.ignite.confi
> >>>>>>>>>>>>>>> guration.DataRegionConfiguration">
> >>>>>>>>>>>>>>>                        
<property name="maxSize"
> value="#{5 *
> >>>>>>>>>>>>>>> 1024 *
> >>>>>>>>>>>>>>> 102 * 1024}"/>
> >>>>>>>>>>>>>>>                     </bean>
> >>>>>>>>>>>>>>>                 </property>
> >>>>>>>>>>>>>>>             </bean>
> >>>>>>>>>>>>>>>         </property>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In other data regions persistence
will be disabled by
> default.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ivan, how to get an instance
to the default region bean and
> >>> change
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> parameter? Obviously, if the goal
is to enable the
> persistence I
> >>>>>>>>>>>>> don’t want
> >>>>>>>>>>>>> to create the default region bean
from scratch.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —
> >>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Oct 2, 2017, at 9:11 AM, Ivan
Rakov <
> ivan.glukos@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Agree with Alexey.
> >>>>>>>>>>>>>> Properties like *defaultDataRegionSize*,
> >>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> >>>>>>>>>>>>>> can confuse users who don't
know that there's such thing as
> >>> default
> >>>>>>>>>>>>>> data
> >>>>>>>>>>>>>> region. They can decide they
are inherited by all data
> regions
> >>>>>>>>>>>>>> where
> >>>>>>>>>>>>>> size
> >>>>>>>>>>>>>> and persistence flag are not
explicitly set.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let's get rid of these properties
and add
> >>>>>>>>>>>>>> *defaultRegionConfiguration*
> >>>>>>>>>>>>>> property with explicit configuration
of default data region.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regarding XML configuration,
changing size or persistence
> flag
> >>> of
> >>>>>>>>>>>>>> default data region will be
just two lines longer (for bean
> >>>>>>>>>>>>>> description):
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         <property name="dataStorageConfiguration">
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>             <bean class="org.apache.ignite.confi
> >>>>>>>>>>>>>>> guration.DataStorageConfiguration">
> >>>>>>>>>>>>>>>                 <property
name="systemCacheInitialSize"
> >>>>>>>>>>>>>>> value="#{100
> >>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>> 1024 * 1024}"/>
> >>>>>>>>>>>>>>>                 <property
name="
> defaultRegionConfiguration">
> >>>>>>>>>>>>>>>                     <bean
class="org.apache.ignite.confi
> >>>>>>>>>>>>>>> guration.DataRegionConfiguration">
> >>>>>>>>>>>>>>>                        
<property name="maxSize"
> value="#{5 *
> >>>>>>>>>>>>>>> 1024 *
> >>>>>>>>>>>>>>> 102 * 1024}"/>
> >>>>>>>>>>>>>>>                     </bean>
> >>>>>>>>>>>>>>>                 </property>
> >>>>>>>>>>>>>>>             </bean>
> >>>>>>>>>>>>>>>         </property>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In other data regions persistence
will be disabled by
> default.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've updated draft in https://issues.apache.org/jira
> >>>>>>>>>>>>>> /browse/IGNITE-6030 with these
changes.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Ivan Rakov
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 02.10.2017 18:35, Denis Magda
wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> To resolve this, I suggest to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> introduce just another field
defaultRegionConfiguration and
> >>> get
> >>>>>>>>>>>>>>>> rid
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> other defaults in DataStorageConfiguration.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Won’t it complicate
the configuration from a Spring XML
> >>> file? I’m
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> an expert in Spring so how
do I get
> defaultRegionConfiguration
> >>>>>>>>>>>>>>> bean
> >>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>> to change any parameter?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> —
> >>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Oct 2, 2017, at 8:30
AM, Alexey Goncharuk <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
wrote:
> >>>>>>>>>>>>>>>> Agree with Vladimir.
If we are to implement this, we would
> >>> either
> >>>>>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>> have a Boolean (non-primitive)
for persistenceEnabled on
> >>>>>>>>>>>>>>>> DataRegionConfiguration,
or introduce an enum for this
> field
> >>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>> is also
> >>>>>>>>>>>>>>>> an overkill. On the
other hand, one can assume that the
> >>> defaults
> >>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> talking about are actually
inherited. To resolve this, I
> >>> suggest
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> introduce just another
field defaultRegionConfiguration
> and
> >>> get
> >>>>>>>>>>>>>>>> rid
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> other defaults in DataStorageConfiguration.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2017-10-02 15:19 GMT+03:00
Ivan Rakov <
> ivan.glukos@gmail.com
> >>>> :
> >>>>>>>>>>>>>>>> Vladimir,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I like your approach
because it's easier to implement.
> >>>>>>>>>>>>>>>>> However, user may
be confused by setting
> >>>>>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> >>>>>>>>>>>>>>>>> flag and seeing
that persistence is not enabled by
> default
> >>> in
> >>>>>>>>>>>>>>>>> custom memory
> >>>>>>>>>>>>>>>>> region. I'll add
clarifying Javadoc at this place.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Ivan Rakov
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 02.10.2017 11:28,
Vladimir Ozerov wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ivan,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I do not think this
is correct approach, because it will
> be
> >>> hard
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> explain, and
you will have to use "Boolean" instead of
> >>>>>>>>>>>>>>>>>> "boolean"
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> DataRegionConfiguration.
I do not think we need default
> >>>>>>>>>>>>>>>>>> "persistence
> >>>>>>>>>>>>>>>>>> enabled" for
all regions. Instead, we should have
> >>> "persistence
> >>>>>>>>>>>>>>>>>> enabled"
> >>>>>>>>>>>>>>>>>> flag for default
region only. It should not be
> propagated
> >>> to
> >>>>>>>>>>>>>>>>>> custom
> >>>>>>>>>>>>>>>>>> regions.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Oct
2, 2017 at 11:14 AM, Ivan Rakov <
> >>>>>>>>>>>>>>>>>> ivan.glukos@gmail.com
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Guys, I think
I got the point now.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Let's check
the final design:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> *DataStorageConfiguration*
will have
> >>>>>>>>>>>>>>>>>>> *isDefaultPersistenceEnabled*
> >>>>>>>>>>>>>>>>>>> property
(default = false), which will be used for
> >>> enabling
> >>>>>>>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> default
data region.
> >>>>>>>>>>>>>>>>>>> *DataRegionConfiguration*
will have
> *isPersistenceEnabled*
> >>>>>>>>>>>>>>>>>>> property,
> >>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>> will be
used for enabling persistence in corresponding
> >>> data
> >>>>>>>>>>>>>>>>>>> region.
If
> >>>>>>>>>>>>>>>>>>> value is
not set, value of
> *DataStorageConfiguration::isD
> >>>>>>>>>>>>>>>>>>> efaultPersistenceEnabled*
> >>>>>>>>>>>>>>>>>>> will be
used by default.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Ivan Rakov
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 02.10.2017
7:49, Dmitriy Setrakyan wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon,
Oct 2, 2017 at 7:46 AM, Denis Magda <
> >>>>>>>>>>>>>>>>>>> dmagda@apache.org>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Oct 1,
2017, at 4:41 AM, Ivan Rakov <
> >>> ivan.glukos@gmail.com
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 1) You're
right. I forgot to include the main flag in
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DataRegionConfiguration
- *isPersistenceEnabled*.
> >>> Persistence
> >>>>>>>>>>>>>>>>>>>>>>
will be
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
enabled globally if at least one memory region has
> this
> >>>>>>>>>>>>>>>>>>>>>>
flag
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
set.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
I’m confused. Why the persistence should be enabled
> >>>>>>>>>>>>>>>>>>>>>
*globally*
> >>>>>>>>>>>>>>>>>>>>>
if the
> >>>>>>>>>>>>>>>>>>>>>
purpose is to have it set for a specific region? If
> it’s
> >>>>>>>>>>>>>>>>>>>>>
enabled for
> >>>>>>>>>>>>>>>>>>>>>
region
> >>>>>>>>>>>>>>>>>>>>>
A only, I don’t want to have it activated for region
> B.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
Yes, you are right. By default the persistence will
> be
> >>>>>>>>>>>>>>>>>>>>>
disabled
> >>>>>>>>>>>>>>>>>>>>>
globally.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
But we should also give users a way to switch the
> >>> default
> >>>>>>>>>>>>>>>>>>>>>
behavior from
> >>>>>>>>>>>>>>>>>>>> in-memory
only (no-persistence) to persistence.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message