samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rayman preet <rayman7...@gmail.com>
Subject Re: Tracing the Samza+YARN startup process
Date Fri, 31 May 2019 16:27:34 GMT
Yes I think your hunch is right. Each container queries the AM over HTTP to
obtain
the jobModel that it is supposed to run. The AM runs a HTTP server usually
on
a dynamically allocated free port on the machine it's running on.
So its possible that a firewall rule blocks the container when it tries to
reach this port
on the AM's machine?

--
thanks
rayman

On Thu, May 30, 2019 at 5:30 PM Malcolm McFarland <mmcfarland@cavulus.com>
wrote:

> Thanks for the image, appreciate you taking the effort to do that! I'm
> still hitting this wall. The AM will launch the container, the container
> will go from "accepted" to "running", but there will be no output from the
> container (I'm piping all of the Samza, org.apache, org.kafka, and our own
> application's logging output to a Kafka topic). During these periods, the
> container will hang out at ~100MB/8GB memory usage and stall. There's no
> error output when this happens; it just kind of stops. My suspicion is that
> our Ops group has a firewall rule up that's interfering with this,or maybe
> just isn't white-listing a port correctly, and if I could identify where
> the application is stalling, it'd probably help to narrow down the
> possibilities.
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
>
> On Thu, May 30, 2019 at 1:39 PM rayman preet <rayman7718@gmail.com> wrote:
>
> > I uploaded the image here:
> > https://www.dropbox.com/s/rv57v165ysp12c5/samza%20flow.png?dl=0
> >
> > Are you still running into this issue?
> > Is there anything in the container's log that shows any
> exceptions/errors.
> >
> > On Wed, May 22, 2019 at 10:15 PM Malcolm McFarland <
> mmcfarland@cavulus.com
> > >
> > wrote:
> >
> > > Hey rayman,
> > >
> > > What it looks like is that the AM has started, the container has
> started,
> > > but, ie, here will be the last messages I see in the Samza logs:
> > >
> > > 2019-05-23T05:10:45.048Z        INFO    Making a request for ANY_HOST
> > > 2019-05-23T05:10:45.057Z        INFO    Starting the container
> allocator
> > > thread
> > > 2019-05-23T05:10:47.098Z        INFO    Received new token for :
> > > <valid_host>:8032
> > > 2019-05-23T05:10:47.102Z        INFO    Container allocated from RM on
> > > <same_valid_host>
> > > 2019-05-23T05:10:47.105Z        INFO    Container allocated from RM on
> > > <same_valid_host>
> > >
> > > At this point, it seems to stall, and no more output is produced.
> > >
> > > Also, I couldn't see you diagram (it's possible my company's email
> > filters
> > > attachments); can I see that on the web anywhere?
> > >
> > > Cheers,
> > > Malcolm
> > >
> > > On Wed, May 22, 2019 at 4:30 PM rayman preet <rayman7718@gmail.com>
> > wrote:
> > >
> > > > Hi Malcolm,
> > > >
> > > > This figure (attached) gives an overview of the flow. Is
> > > > this something you were looking for?
> > > >
> > > > Also, by "don't fully start up" do you mean that
> > > > applications are missing some containers (but the ApplicationMaster
> is
> > > > running)?
> > > > Or the application is missing entirely.
> > > >
> > > > --
> > > > thanks
> > > > rayman
> > > > [image: Samza Job Launch Sequence.png]
> > > >
> > > > On Tue, May 21, 2019 at 3:58 PM Malcolm McFarland <
> > > mmcfarland@cavulus.com>
> > > > wrote:
> > > >
> > > >> Hey Folks,
> > > >>
> > > >> I'm still trying to pin down why these applications are sometimes
> not
> > > >> starting. Everything looks fine in the YARN web UI and in the
> > > >> immediately available logs, but the applications don't always fully
> > > >> start up. Does anybody have a rundown about how to trace the Samza
> > > >> startup process on a YARN cluster, from Accepted status, to
> > > >> localization, to the application master startup, to the actual
> > > >> application's startup?
> > > >>
> > > >> Cheers,
> > > >> Malcolm
> > > >>
> > > >> --
> > > >> Malcolm McFarland
> > > >> Cavulus
> > > >>
> > > >>
> > > >> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > >> unauthorized or improper disclosure, copying, distribution, or use
> of
> > > >> the contents of this message is prohibited. The information
> contained
> > > >> in this message is intended only for the personal and confidential
> use
> > > >> of the recipient(s) named above. If you have received this message
> in
> > > >> error, please notify the sender immediately and delete the original
> > > >> message.
> > > >>
> > > >
> > > >
> > > > --
> > > > thanks
> > > > rayman
> > > >
> > >
> > >
> > > --
> > > Malcolm McFarland
> > > Cavulus
> > > 1-800-760-6915
> > > mmcfarland@cavulus.com
> > >
> > >
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > unauthorized or improper disclosure, copying, distribution, or use of
> the
> > > contents of this message is prohibited. The information contained in
> this
> > > message is intended only for the personal and confidential use of the
> > > recipient(s) named above. If you have received this message in error,
> > > please notify the sender immediately and delete the original message.
> > >
> >
> >
> > --
> > thanks
> > rayman
> >
>


-- 
thanks
rayman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message