spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Spark History Server can't read event logs
Date Wed, 07 Jan 2015 15:45:58 GMT

When I run jobs and save the event logs, they are saved with the permissions of the unix user
and group that ran the spark job. The history server is run as a service account and therefore
can’t read the files:

Extract from the History server logs:

2015-01-07 15:37:24,3021 ERROR Client fs/client/fileclient/cc/ Thread: 1183
User does not have access to open file /apps/spark/historyserver/logs/spark-1420644521194
15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark event log /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1 Open failed for file: /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1,
error: Permission denied (13)

Is there a setting which I can change that allows the files to be world readable or at least
by the account running the history server? Currently, the job appears in the History Sever
UI but only states ‘<Not Started>’.


This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail:

View raw message