ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: [IGNITE-5717] improvements of MemoryPolicy default size
Date Wed, 02 Aug 2017 07:33:55 GMT
Dima,

Probably folks who worked closely with storage know why. However, even
increments will not help us. Most developers work on laptops. Common laptop
has 8-16 Gb RAM today. OS and working programs typically consume about a
half. So, for example, I have 8Gb total, 4Gb consumed, 4Gb free. And Ignite
will try to allocate up to 6.5Gb. Even if it is done incrementally, once I
tried to upload more than 4Gb of data, my laptop will hang silently, and
tomorrow I would tell my fellow engineers "Tried Ignite, it hanged, so I
have to reboot".

The very problem is that before being started once on production
environment, Ignite will typically be started hundred times on developer's
environment. I think that default should be ~10% of total RAM.

On Wed, Aug 2, 2017 at 10:21 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> On Wed, Aug 2, 2017 at 7:27 AM, Vladimir Ozerov <vozerov@gridgain.com>
> wrote:
>
> > Please see original Sergey's message - when persistence is enabled,
> memory
> > is not allocated incrementally, maxSize is used.
> >
>
> Why?
>
>
> > Default settings must allow for normal work on developer's environment.
> >
>
> Agree, but why not in increments?
>
>
> >
> > ср, 2 авг. 2017 г. в 1:10, Denis Magda <dmagda@apache.org>:
> >
> > > > Why not allocate in increments automatically?
> > >
> > > This is exactly how the allocation works right now. The memory will
> grow
> > > incrementally until the max size is reached (80% of RAM by default).
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 1, 2017, at 3:03 PM, dsetrakyan@apache.org wrote:
> > > >
> > > > Vova, 1GB seems a bit too small for me, and frankly i do not want t o
> > > guess. Why not allocate in increments automatically?
> > > >
> > > > ⁣D.​
> > > >
> > > > On Aug 1, 2017, 11:03 PM, at 11:03 PM, Vladimir Ozerov <
> > > vozerov@gridgain.com> wrote:
> > > >> Denis,
> > > >> No doubts you haven't heard about it - AI 2.1 with persistence, when
> > > >> 80% of
> > > >> RAM is allocated right away, was released several days ago. How do
> you
> > > >> think, how many users tried it already?
> > > >>
> > > >> Guys,
> > > >> Do you really think allocating 80% of available RAM is a normal
> thing?
> > > >> Take
> > > >> your laptop and check how many available RAM you have right now. Do
> > you
> > > >> fit
> > > >> to remaining 20%? If not, then running AI with persistence with all
> > > >> defaults will bring your machine down. This is insane. We shold
> > > >> allocate no
> > > >> more than 1Gb, so that user can play with it without any problems.
> > > >>
> > > >> On Tue, Aug 1, 2017 at 10:26 PM, Denis Magda <dmagda@apache.org>
> > wrote:
> > > >>
> > > >>> My vote goes for option #1 too. I don’t think that 80% is too
> > > >> aggressive
> > > >>> to bring it down.
> > > >>>
> > > >>> IGNITE-5717 was created to fix the issue of the 80% RAM allocation
> on
> > > >> 64
> > > >>> bit systems when Ignite works on top of 32 bit JVM. I’ve not
heard
> of
> > > >> any
> > > >>> other complaints in regards the default allocation size.
> > > >>>
> > > >>> —
> > > >>> Denis
> > > >>>
> > > >>>> On Aug 1, 2017, at 10:58 AM, dsetrakyan@apache.org wrote:
> > > >>>>
> > > >>>> I prefer option #1.
> > > >>>>
> > > >>>> ⁣D.​
> > > >>>>
> > > >>>> On Aug 1, 2017, 11:20 AM, at 11:20 AM, Sergey Chugunov <
> > > >>> sergey.chugunov@gmail.com> wrote:
> > > >>>>> Folks,
> > > >>>>>
> > > >>>>> I would like to get back to the question about MemoryPolicy
> > > >> maxMemory
> > > >>>>> defaults.
> > > >>>>>
> > > >>>>> Although MemoryPolicy may be configured with initial and
> maxMemory
> > > >>>>> settings, when persistence is used MemoryPolicy always
allocates
> > > >>>>> maxMemory
> > > >>>>> size for performance reasons.
> > > >>>>>
> > > >>>>> As default size of maxMemory is 80% of physical memory
it causes
> > > >> OOME
> > > >>>>> exceptions of 32 bit platforms (either on OS or JVM level)
and
> > > >> hurts
> > > >>>>> performance in setups when multiple Ignite nodes are started
on
> > > >> the
> > > >>>>> same
> > > >>>>> physical server.
> > > >>>>>
> > > >>>>> I suggest to rethink these defaults and switch to other
options:
> > > >>>>>
> > > >>>>> - Check whether platform is 32 or 64 bits and adapt defaults.
In
> > > >> this
> > > >>>>> case we still need to address the issue with multiple
nodes on
> one
> > > >>>>> machine
> > > >>>>> even on 64 bit systems.
> > > >>>>>
> > > >>>>> - Lower defaults for maxMemory and allocate, for instance,
> > > >> max(0.3 *
> > > >>>>> availableMemory, 1Gb).
> > > >>>>> This option allows us to solve all issues with starting
on 32 bit
> > > >>>>> platforms and reduce instability with multiple nodes on
the same
> > > >>>>> machine.
> > > >>>>>
> > > >>>>>
> > > >>>>> Thoughts and/or other options?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Sergey.
> > > >>>
> > > >>>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message