samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Malcolm McFarland <mmcfarl...@cavulus.com>
Subject Re: AM resource needs
Date Fri, 24 May 2019 17:34:34 GMT
Thanks Jagadish, just wanted to verify that this wasn't an obvious thing I
was missing. I'm also setting the AM's heap size explicitly with the
yarn.am.opts configuration parameter; in this case, I'm allowing 512MB for
the JVM on top of my heap for both the AM and the container (ie,
yarn.am.container.memory.mb=1536, yarn.am.opts=-Xmx1024m); does that sound
reasonable?

Cheers,
Malcolm McFarland
Cavulus


This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.

Malcolm McFarland
Cavulus


This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.


On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman <
jagadish1989@gmail.com> wrote:

> No, the default setup should be sufficient - the number of tasks should
> have no significant impact on AM memory/resources.
>
> If you run out-of-memory, you can of course increase yarn.am.memory.mb.
>
> On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <mmcfarland@cavulus.com
> >
> wrote:
>
> > Hey folks,
> >
> > Are there any guidelines for how to provision an Application Master in
> > relation to the number of StreamTask instances it will be managing? Ie,
> are
> > there different memory, CPU, and thread-count figures for 100S
> StreamTasks
> > vs 1000, vs 10000?
> >
> > Cheers,
> > Malcolm McFarland
> > Cavulus
> >
> >
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > unauthorized or improper disclosure, copying, distribution, or use of the
> > contents of this message is prohibited. The information contained in this
> > message is intended only for the personal and confidential use of the
> > recipient(s) named above. If you have received this message in error,
> > please notify the sender immediately and delete the original message.
> >
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message