flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Yao <g...@data-artisans.com>
Subject Re: Access logs for a running Flink app in YARN cluster
Date Fri, 13 Apr 2018 19:13:35 GMT
Hi,

I see two options:

1. You can login to the slave machines, which run the NodeManagers, and
access
the container logs. The path of the container logs can be configured in
yarn-site.xml with the key yarn.nodemanager.log-dirs. In my tests with EMR,
the
logs are stored at /var/log/hadoop-yarn/containers/ by default. For example:

  sudo ls
/var/log/hadoop-yarn/containers/application_1523645835493_0002/container_1523645835493_0002_01_000002
  taskmanager.err  taskmanager.log  taskmanager.out

2. You can download the logs via HTTP from Flink:

  http://host:port/jobmanager/log
  http://host:port/taskmanagers/<taskmanager-id>/log

To get a list of taskmanagers:

  http://host:port/taskmanagers

Best,
Gary

On Fri, Apr 13, 2018 at 2:09 AM, Tao Xia <tao@udacity.com> wrote:

> Any good way to get access container logs from a running Flink app in YARN
> cluster in EMR?
> You can view the logs through YARN UI. But cannot programmatically access
> it and send to other services.
> The log aggregator only runs when the application finishes or a minimum
> 3600 secs copy. Any way we can get the logs more frequently?
>

Mime
View raw message