ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Павлухин Иван <vololo...@gmail.com>
Subject Re: Pre-touch for Ignite off-heap memory
Date Tue, 18 Dec 2018 13:01:24 GMT
Stan,

In one of your emails you wrote why you need pre-touch:
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical
page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself

As far as I see only c) can be satisfied if lazy region initialization
is enabled. Am I missing something?
пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov <stanlukyanov@gmail.com>:
>
> I don’t think these two issues require to be solved together, although I agree there
is some connection.
>
>
>
> I guess we agree that pre-touch should a be off by default.
>
> Similarly, lazy data region initialization from IGNITE-9113 can be on by default.
>
> We can have two boolean properties controlling each feature, e.g.
>
>     IGNITE_EAGER_DATA_REGION_INITIALIZATION
>
>     IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
>
> Setting both to true will make sure that all memory is committed as early as possible.
>
> Different combinations will allow for all 4 variants you’ve mentioned. I can see a
use case for each one:
>
> - lazy region init + no pre-touch (default) – easier configuration (at least for simple
use cases without a lot of regions and node filters) and faster startup
>
> - lazy region init + pre-touch – allows to reuse server config on a client (the goal
of the IGNITE-9113) but still allows to fail-fast on cache creation
>
> - eager region init + no pre-touch – for cases when you want to eagerly allocate memory
but don’t care about committing physical pages; e.g. if you have overcommit disabled then
you don’t need to pre-touch everything to make sure the memory is there
>
> - eager region init + pre-touch – to fail as early as possible if there is not enough
RAM
>
>
>
> Stan
>
>
>
> From: Nikolay Izhikov
> Sent: 14 декабря 2018 г. 14:42
> To: dev; vololo100@gmail.com; stanlukyanov@gmail.com
> Subject: Re: Pre-touch for Ignite off-heap memory
>
>
>
> Bump.
>
>
>
> Stanislav, Ivan, can you answer my questions?
>
> Let's discuss these improvements.
>
>
>
> ср, 12 дек. 2018 г. в 22:14, Павлухин Иван <vololo100@gmail.com>:
>
>
>
> > Hi Stan,
>
> >
>
> > As I participated in discussion in current thread I would like to
>
> > leave a comment.
>
> >
>
> > Your concerns looks clear for me and if you believe that pre-touch
>
> > will help product users then I have nothing against it.
>
> > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <nizhikov@apache.org>:
>
> > >
>
> > > Hello, Stanislav.
>
> > >
>
> > > As far as I can see, we have controversal ticket based on previous
>
> > dev-list discussion:
>
> > >
>
> > > IGNITE-9113 - Allocate memory for a data region when first cache
>
> > assigned to this region is created [1]
>
> > >
>
> > > I planned to implement it soon.
>
> > > Looks like we should have several options of memory(data regions)
>
> > allocation:
>
> > >
>
> > >         - allocate all on startup (AFAIK this is how current
>
> > implementation behave)
>
> > >         - allocate all on startup AND pre touch.
>
> > >         - allocate specific data region for first assignment.
>
> > >         - allocate specific data region for first assignment AND pre
>
> > touch.
>
> > >
>
> > > What do you think?
>
> > >
>
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> > >
>
> > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
>
> > > > Igniters,
>
> > > >
>
> > > > What is being suggested here is an Ignite off-heap’s version of Java’s
>
> > -XX:+AlwaysPreTouch.
>
> > > > The latter is known to be used to guarantee that the committed memory
>
> > is backed by physical RAM.
>
> > > > This ensures that
>
> > > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
>
> > for physical page allocation, possible contention with page cache, etc)
>
> > > > b) JVM fails fast if the Xmx is greater than available RAM
>
> > > > c) New processes will not be able to claim the memory JVM took for
>
> > itself
>
> > > >
>
> > > > Currently one can’t get the same benefits for Ignite because we use
>
> > off-heap as well as heap.
>
> > > > So, we can implement a similar feature for Ignite – and make sure the
>
> > users can get all the memory pre-touch benefits if they want.
>
> > > > Of course, it impacts startup time so we should enable it by a
>
> > configuration property (I’d suggest a system property for now).
>
> > > >
>
> > > > Are there any objections to implementing this?
>
> > > >
>
> > > > Thanks,
>
> > > > Stan
>
> > > >
>
> > > > From: Павлухин Иван
>
> > > > Sent: 31 октября 2018 г. 12:50
>
> > > > To: dev@ignite.apache.org
>
> > > > Subject: Re: Pre-touch for Ignite off-heap memory
>
> > > >
>
> > > > Hi,
>
> > > >
>
> > > > I did an experiment described above. I created a patch with pre-touch
>
> > [1]
>
> > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
>
> > > > Ignite with equal values for initial and max sizes for each data
>
> > region.
>
> > > > I did several runs. I observed JVM crash dumps [2], [3]. Also it is
>
> > easy
>
> > > > to observe JVM OOM-killed.
>
> > > >
>
> > > > [1] https://github.com/apache/ignite/pull/5220
>
> > > > [2]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
>
> > > > [3]
>
> > > >
>
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
>
> > > >
>
> > > > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vololo100@gmail.com>:
>
> > > >
>
> > > > > Hi guys,
>
> > > > >
>
> > > > > I am not aware that it is possible to run JVM in "allocation-free"
>
> > fashion.
>
> > > > > If you know that it is possible please share it. As I know JVM
>
> > allocates
>
> > > > > memory out of garbage collectable area for internal purposes like
>
> > JIT,
>
> > > > > GC itself. Also native built-it code can request memory allocation
>
> > from OS,
>
> > > > > e.g. zip facilities. If we imagine that system is running under
>
> > memory
>
> > > > > allocation
>
> > > > > which is close to a limit then even small allocation request can
>
> > fail and
>
> > > > > lead
>
> > > > > to OOM killing.
>
> > > > >
>
> > > > > But I think that a simple and useful thing that could be done first
>
> > is
>
> > > > > making
>
> > > > > an experiment. As Andrey mentioned
>
> > > > > > AFAIK, Ignite always pre-touch first region. So, you can try
to set
>
> > > > >
>
> > > > > region
>
> > > > > > MAX size equal to MIN and get region allocated on node start.
>
> > > > >
>
> > > > > If it is so then it should not be hard to launch Ignite and observe
>
> > it
>
> > > > > running
>
> > > > > very close to OS memory limit with Java heap and Ignite off-heap
>
> > > > > pre-touched.
>
> > > > > After that one could check whether it is possible to observe Ignite
>
> > OOM
>
> > > > > killed.
>
> > > > > Let's say my bet is that it is relatively easy to catch OOM here.
>
> > > > >
>
> > > > > What do you think?
>
> > > > >
>
> > > > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yzhdanov@apache.org>:
>
> > > > >
>
> > > > > > Andrey,
>
> > > > > >
>
> > > > > > Probability of a OOM kill will be much lower if offheap is
>
> > pretouched.
>
> > > > > > What
>
> > > > > > do you mean by JVM internal needs? In my understanding if user
>
> > enables
>
> > > > > > option to pretouch heap and fixes the heap to prevent jvm
>
> > releasing memory
>
> > > > > > back to OS, then OOM killing is very unlikely.
>
> > > > > >
>
> > > > > > I would agree that pretouch for offheap may be helpful in many
>
> > cases.
>
> > > > > >
>
> > > > > > --Yakov
>
> > > > > >
>
> > > > >
>
> > > > >
>
> > > > > --
>
> > > > > Best regards,
>
> > > > > Ivan Pavlukhin
>
> > > > >
>
> > > >
>
> > > >
>
> >
>
> >
>
> >
>
> > --
>
> > Best regards,
>
> > Ivan Pavlukhin
>
> >
>
>



-- 
Best regards,
Ivan Pavlukhin

Mime
View raw message