spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Spark History Server can't read event logs
Date Wed, 07 Jan 2015 18:09:30 GMT
The Spark code generates the log directory with "770" permissions. On
top of that you need to make sure of two things:

- all directories up to /apps/spark/historyserver/logs/ are readable
by the user running the history server
- the user running the history server belongs to the group that owns
/apps/spark/historyserver/logs/

I think the code could be more explicitly about setting the group of
the generated log directories and files, but if you follow the two
rules above things should work. Also, I recommend setting
/apps/spark/historyserver/logs/ itself to "1777" so that any user can
generate logs, but only the owner (or a superuser) can delete them.



On Wed, Jan 7, 2015 at 7:45 AM,  <michael.england@nomura.com> wrote:
> Hi,
>
>
>
> When I run jobs and save the event logs, they are saved with the permissions
> of the unix user and group that ran the spark job. The history server is run
> as a service account and therefore can’t read the files:
>
>
>
> Extract from the History server logs:
>
>
>
> 2015-01-07 15:37:24,3021 ERROR Client fs/client/fileclient/cc/client.cc:1009
> Thread: 1183 User does not have access to open file
> /apps/spark/historyserver/logs/spark-1420644521194
>
> 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark event
> log /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1
>
> org.apache.hadoop.security.AccessControlException: Open failed for file:
> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error:
> Permission denied (13)
>
>
>
> Is there a setting which I can change that allows the files to be world
> readable or at least by the account running the history server? Currently,
> the job appears in the History Sever UI but only states ‘<Not Started>’.
>
>
>
> Thanks,
>
> Michael
>
>
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from taking
> action on the basis of information in this e-mail and must contact the
> sender immediately, delete this e-mail (and all attachments) and destroy any
> hard copies. Nomura will not accept responsibility or liability for the
> accuracy or completeness of, or the presence of any virus or disabling code
> in, this e-mail. If verification is sought please request a hard copy. Any
> reference to the terms of executed transactions should be treated as
> preliminary only and subject to formal written confirmation by Nomura.
> Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a
> reference to any entity in the Nomura Holdings, Inc. group. Please read our
> Electronic Communications Legal Notice which forms part of this e-mail:
> http://www.Nomura.com/email_disclaimer.htm



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message