samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Malcolm McFarland <mmcfarl...@cavulus.com>
Subject Re: Tracing the Samza+YARN startup process
Date Fri, 31 May 2019 00:30:19 GMT
Thanks for the image, appreciate you taking the effort to do that! I'm
still hitting this wall. The AM will launch the container, the container
will go from "accepted" to "running", but there will be no output from the
container (I'm piping all of the Samza, org.apache, org.kafka, and our own
application's logging output to a Kafka topic). During these periods, the
container will hang out at ~100MB/8GB memory usage and stall. There's no
error output when this happens; it just kind of stops. My suspicion is that
our Ops group has a firewall rule up that's interfering with this,or maybe
just isn't white-listing a port correctly, and if I could identify where
the application is stalling, it'd probably help to narrow down the
possibilities.

Cheers,
Malcolm McFarland
Cavulus


This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.


On Thu, May 30, 2019 at 1:39 PM rayman preet <rayman7718@gmail.com> wrote:

> I uploaded the image here:
> https://www.dropbox.com/s/rv57v165ysp12c5/samza%20flow.png?dl=0
>
> Are you still running into this issue?
> Is there anything in the container's log that shows any exceptions/errors.
>
> On Wed, May 22, 2019 at 10:15 PM Malcolm McFarland <mmcfarland@cavulus.com
> >
> wrote:
>
> > Hey rayman,
> >
> > What it looks like is that the AM has started, the container has started,
> > but, ie, here will be the last messages I see in the Samza logs:
> >
> > 2019-05-23T05:10:45.048Z        INFO    Making a request for ANY_HOST
> > 2019-05-23T05:10:45.057Z        INFO    Starting the container allocator
> > thread
> > 2019-05-23T05:10:47.098Z        INFO    Received new token for :
> > <valid_host>:8032
> > 2019-05-23T05:10:47.102Z        INFO    Container allocated from RM on
> > <same_valid_host>
> > 2019-05-23T05:10:47.105Z        INFO    Container allocated from RM on
> > <same_valid_host>
> >
> > At this point, it seems to stall, and no more output is produced.
> >
> > Also, I couldn't see you diagram (it's possible my company's email
> filters
> > attachments); can I see that on the web anywhere?
> >
> > Cheers,
> > Malcolm
> >
> > On Wed, May 22, 2019 at 4:30 PM rayman preet <rayman7718@gmail.com>
> wrote:
> >
> > > Hi Malcolm,
> > >
> > > This figure (attached) gives an overview of the flow. Is
> > > this something you were looking for?
> > >
> > > Also, by "don't fully start up" do you mean that
> > > applications are missing some containers (but the ApplicationMaster is
> > > running)?
> > > Or the application is missing entirely.
> > >
> > > --
> > > thanks
> > > rayman
> > > [image: Samza Job Launch Sequence.png]
> > >
> > > On Tue, May 21, 2019 at 3:58 PM Malcolm McFarland <
> > mmcfarland@cavulus.com>
> > > wrote:
> > >
> > >> Hey Folks,
> > >>
> > >> I'm still trying to pin down why these applications are sometimes not
> > >> starting. Everything looks fine in the YARN web UI and in the
> > >> immediately available logs, but the applications don't always fully
> > >> start up. Does anybody have a rundown about how to trace the Samza
> > >> startup process on a YARN cluster, from Accepted status, to
> > >> localization, to the application master startup, to the actual
> > >> application's startup?
> > >>
> > >> Cheers,
> > >> Malcolm
> > >>
> > >> --
> > >> Malcolm McFarland
> > >> Cavulus
> > >>
> > >>
> > >> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > >> unauthorized or improper disclosure, copying, distribution, or use of
> > >> the contents of this message is prohibited. The information contained
> > >> in this message is intended only for the personal and confidential use
> > >> of the recipient(s) named above. If you have received this message in
> > >> error, please notify the sender immediately and delete the original
> > >> message.
> > >>
> > >
> > >
> > > --
> > > thanks
> > > rayman
> > >
> >
> >
> > --
> > Malcolm McFarland
> > Cavulus
> > 1-800-760-6915
> > mmcfarland@cavulus.com
> >
> >
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > unauthorized or improper disclosure, copying, distribution, or use of the
> > contents of this message is prohibited. The information contained in this
> > message is intended only for the personal and confidential use of the
> > recipient(s) named above. If you have received this message in error,
> > please notify the sender immediately and delete the original message.
> >
>
>
> --
> thanks
> rayman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message