flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sachin Goel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2732) Add access to the TaskManagers' log file and out file in the web dashboard.
Date Fri, 16 Oct 2015 11:46:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960549#comment-14960549

Sachin Goel commented on FLINK-2732:

I realize there is no communication required between TMs, which is why I mentioned it. Opening
a port for log file access on the TM does not break any fault-tolerance guarantees. Maybe
it was a misplaced comment. 

As for the firewall, it is not the user's browser which will access the log, rather the Web
frontend itself, which should be able to do that since it lives on the same host as the job
manager. What my design entails is this:

User's browser ---> /logs?host=<akka_url_to_actor>&port=<blob_port_for_log_access>&type=<log|out>
Web frontend: handles requests of this type with a handler which opens a connection to the
blob server and fetches the desired file. [Let's not call it the blob server, rather the log
server. It serves files by name, not by blob key.]

As for the resource consumption, I think it is a trade-off. We can either have an additional
network hop, or run a thread on the TMs, which doesn't maintain any data structures, and uses
minimal memory to establish the server. [This is my primary concern too which is why I'd already
mentioned that someone else should comment on this. :)]

Furthermore, since in the future, the web frontend won't be re-directing to the leading job
manager, the web frontend which the user is using will need to fetch the logs of the leading
job manager over the network. This design will allow for easy log and stdout access in that
case too. It also abstracts out the Log and Stdout service from the rest of the components,
and in my opinion, is a cleaner design.

> Add access to the TaskManagers' log file and out file in the web dashboard.
> ---------------------------------------------------------------------------
>                 Key: FLINK-2732
>                 URL: https://issues.apache.org/jira/browse/FLINK-2732
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Webfrontend
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Martin Liesenberg
>             Fix For: 0.10
> Add access to the TaskManagers' log file and out file in the web dashboard.
> This needs addition on the server side, as the log files need to be transferred   to
the JobManager via the blob server.

This message was sent by Atlassian JIRA

View raw message