spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <michael.engl...@nomura.com>
Subject RE: Spark History Server can't read event logs
Date Fri, 09 Jan 2015 09:03:08 GMT
Hi Marcelo,

On MapR, the mapr user can read the files using the NFS mount, however using the normal hadoop
fs -cat /... command, I get permission denied. As the history server is pointing to a location
on mapfs, not the NFS mount, I'd imagine the Spark history server is trying to read the files
using the hadoop api and therefore the permissions cause issues here.

Thanks,
Michael


-----Original Message-----
From: Marcelo Vanzin [mailto:vanzin@cloudera.com] 
Sent: 08 January 2015 19:23
To: England, Michael (IT/UK)
Cc: user@spark.apache.org
Subject: Re: Spark History Server can't read event logs

Sorry for the noise; but I just remembered you're actually using MapR (and not HDFS), so maybe
the "3777" trick could work...

On Thu, Jan 8, 2015 at 10:32 AM, Marcelo Vanzin <vanzin@cloudera.com> wrote:
> Nevermind my last e-mail. HDFS complains about not understanding "3777"...
>
> On Thu, Jan 8, 2015 at 9:46 AM, Marcelo Vanzin <vanzin@cloudera.com> wrote:
>> Hmm. Can you set the permissions of "/apps/spark/historyserver/logs"
>> to 3777? I'm not sure HDFS respects the group id bit, but it's worth 
>> a try. (BTW that would only affect newly created log directories.)
>>
>> On Thu, Jan 8, 2015 at 1:22 AM,  <michael.england@nomura.com> wrote:
>>> Hi Vanzin,
>>>
>>> I am using the MapR distribution of Hadoop. The history server logs are created
by a job with the permissions:
>>>
>>> drwxrwx---   - <myusername> <mygroup>               2 2015-01-08
09:14 /apps/spark/historyserver/logs/spark-1420708455212
>>>
>>> However, the permissions of the higher directories are mapr:mapr and the user
that runs Spark in our case is a unix ID called mapr (in the mapr group). Therefore, this
can't read my job event logs as shown above.
>>>
>>>
>>> Thanks,
>>> Michael
>>>
>>>
>>> -----Original Message-----
>>> From: Marcelo Vanzin [mailto:vanzin@cloudera.com]
>>> Sent: 07 January 2015 18:10
>>> To: England, Michael (IT/UK)
>>> Cc: user@spark.apache.org
>>> Subject: Re: Spark History Server can't read event logs
>>>
>>> The Spark code generates the log directory with "770" permissions. On top of
that you need to make sure of two things:
>>>
>>> - all directories up to /apps/spark/historyserver/logs/ are readable 
>>> by the user running the history server
>>> - the user running the history server belongs to the group that owns 
>>> /apps/spark/historyserver/logs/
>>>
>>> I think the code could be more explicitly about setting the group of the generated
log directories and files, but if you follow the two rules above things should work. Also,
I recommend setting /apps/spark/historyserver/logs/ itself to "1777" so that any user can
generate logs, but only the owner (or a superuser) can delete them.
>>>
>>>
>>>
>>> On Wed, Jan 7, 2015 at 7:45 AM,  <michael.england@nomura.com> wrote:
>>>> Hi,
>>>>
>>>>
>>>>
>>>> When I run jobs and save the event logs, they are saved with the 
>>>> permissions of the unix user and group that ran the spark job. The 
>>>> history server is run as a service account and therefore can’t read the
files:
>>>>
>>>>
>>>>
>>>> Extract from the History server logs:
>>>>
>>>>
>>>>
>>>> 2015-01-07 15:37:24,3021 ERROR Client
>>>> fs/client/fileclient/cc/client.cc:1009
>>>> Thread: 1183 User does not have access to open file
>>>> /apps/spark/historyserver/logs/spark-1420644521194
>>>>
>>>> 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing 
>>>> Spark event log
>>>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1
>>>>
>>>> org.apache.hadoop.security.AccessControlException: Open failed for file:
>>>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error:
>>>> Permission denied (13)
>>>>
>>>>
>>>>
>>>> Is there a setting which I can change that allows the files to be 
>>>> world readable or at least by the account running the history server?
>>>> Currently, the job appears in the History Sever UI but only states ‘<Not
Started>’.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Michael
>>>>
>>>>
>>>> This e-mail (including any attachments) is private and 
>>>> confidential, may contain proprietary or privileged information and 
>>>> is intended for the named
>>>> recipient(s) only. Unintended recipients are strictly prohibited 
>>>> from taking action on the basis of information in this e-mail and 
>>>> must contact the sender immediately, delete this e-mail (and all
>>>> attachments) and destroy any hard copies. Nomura will not accept 
>>>> responsibility or liability for the accuracy or completeness of, or 
>>>> the presence of any virus or disabling code in, this e-mail. If 
>>>> verification is sought please request a hard copy. Any reference to 
>>>> the terms of executed transactions should be treated as preliminary only
and subject to formal written confirmation by Nomura.
>>>> Nomura reserves the right to retain, monitor and intercept e-mail 
>>>> communications through its networks (subject to and in accordance 
>>>> with applicable laws). No confidentiality or privilege is waived or 
>>>> lost by Nomura by any mistransmission of this e-mail. Any reference 
>>>> to "Nomura" is a reference to any entity in the Nomura Holdings, Inc.
>>>> group. Please read our Electronic Communications Legal Notice which forms
part of this e-mail:
>>>> http://www.Nomura.com/email_disclaimer.htm
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>>
>>> This e-mail (including any attachments) is private and confidential, 
>>> may contain proprietary or privileged information and is intended 
>>> for the named recipient(s) only. Unintended recipients are strictly 
>>> prohibited from taking action on the basis of information in this 
>>> e-mail and must contact the sender immediately, delete this e-mail 
>>> (and all attachments) and destroy any hard copies. Nomura will not 
>>> accept responsibility or liability for the accuracy or completeness 
>>> of, or the presence of any virus or disabling code in, this e-mail. 
>>> If verification is sought please request a hard copy. Any reference 
>>> to the terms of executed transactions should be treated as 
>>> preliminary only and subject to formal written confirmation by 
>>> Nomura. Nomura reserves the right to retain, monitor and intercept 
>>> e-mail communications through its networks (subject to and in 
>>> accordance with applicable laws). No confidentiality or privilege is 
>>> waived or lost by Nomura by any mistransmission of this e-mail. Any 
>>> reference to "Nomura" is a reference to any entity in the Nomura 
>>> Holdings, Inc. group. Please read our Electronic Communications 
>>> Legal Notice which forms part of this e-mail: 
>>> http://www.Nomura.com/email_disclaimer.htm
>>>
>>
>>
>>
>> --
>> Marcelo
>
>
>
> --
> Marcelo



--
Marcelo


This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm

Mime
View raw message