samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <>
Subject Re: Java Opts and Max Heap
Date Tue, 10 Mar 2015 16:16:52 GMT
Hey Jordan,

There are two settings you need to care about when changing memory. One is
yarn.container.memory.mb. The other is the task.opts setting, which will
allow you to specify a custom -Xmx parameter.

If you set the yarn.container.memory.mb to 4000 (4GB), as you suggest, the
JAVA_OPTS check still takes affect, and your Xmx will remain 768MB. If you
wish to increase your heap, you'll need to set both
yarn.container.memory.mb *and* the task.opts setting to be something like
task.opts=-Xmx3g. This will tell YARN to kill any container that uses > 4GB
of physical memory on the machine. The Xmx tells the container not to use
more than 3GB of heap.

It's a bit confusing, but there's a reason for this. The problem we have is
that we (Samza) don't know how much off-heap/non-JVM memory you're going to
use in your container. YARN only pays attention to the amount of physical
memory used by a process. With Java, you can set the heap, but there's also
permgen, JVM, JNI libraries, and off-heap memory usage. All of these
contribute to the physical memory usage that YARN cares about, but are
outside the JVM heap. This means that we can't just use one memory setting
for both YARN and Java. We have to have two.


On Tue, Mar 10, 2015 at 1:13 AM, Jordan Shaw <> wrote:

> Hey Everyone,
> This I have a question somewhat related to SAMZA-109 and this line in
> # Check if a max-heap size is specified. If not - set a 768M heap [[
> $JAVA_OPTS != *-Xmx* ]] && JAVA_OPTS="$JAVA_OPTS -Xmx768M"
> If I were to set the container.memory.mb for yarn to 4GB (
> yarn.container.memory.mb = 4096) the above JAVA_OPTS check would cause this
> to be ignored right? I can understand why preventing the Heap from going
> crazy is important in these long running jobs but to me this might cause
> some confusion especially when trying to debug a thrashing GC and Java
> isn't committing the memory amount set in yarn.container.memory.mb.
> I was trying to follow why this was decided on in SAMZA-109 but nothing
> stuck out to me. Would it make more sense to just not specifying a max heap
> and letting Yarn kill the job if it goes over it's specified allotment
> and/or let the user explicitly set these opts?  Thanks!
> - Jordan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message