samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: AM resource needs
Date Sat, 25 May 2019 07:29:18 GMT
Yup, if that doesn't work for any reason, you can increase memory in 1G
increments.
Ensure that you have some head-room on top of your Xmx to account for native
memory

On Fri, May 24, 2019 at 10:34 AM Malcolm McFarland <mmcfarland@cavulus.com>
wrote:

> Thanks Jagadish, just wanted to verify that this wasn't an obvious thing I
> was missing. I'm also setting the AM's heap size explicitly with the
> yarn.am.opts configuration parameter; in this case, I'm allowing 512MB for
> the JVM on top of my heap for both the AM and the container (ie,
> yarn.am.container.memory.mb=1536, yarn.am.opts=-Xmx1024m); does that sound
> reasonable?
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
>
> On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman <
> jagadish1989@gmail.com> wrote:
>
> > No, the default setup should be sufficient - the number of tasks should
> > have no significant impact on AM memory/resources.
> >
> > If you run out-of-memory, you can of course increase yarn.am.memory.mb.
> >
> > On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <
> mmcfarland@cavulus.com
> > >
> > wrote:
> >
> > > Hey folks,
> > >
> > > Are there any guidelines for how to provision an Application Master in
> > > relation to the number of StreamTask instances it will be managing? Ie,
> > are
> > > there different memory, CPU, and thread-count figures for 100S
> > StreamTasks
> > > vs 1000, vs 10000?
> > >
> > > Cheers,
> > > Malcolm McFarland
> > > Cavulus
> > >
> > >
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > unauthorized or improper disclosure, copying, distribution, or use of
> the
> > > contents of this message is prohibited. The information contained in
> this
> > > message is intended only for the personal and confidential use of the
> > > recipient(s) named above. If you have received this message in error,
> > > please notify the sender immediately and delete the original message.
> > >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>


-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message