hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Container authenticate failed after YARN-961
Date Sat, 14 Dec 2013 02:20:07 GMT
Maybe My yarn was configured incorrect.
in the NM log:
2013-12-14 10:11:48,660 DEBUG
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
health-status : true,
2013-12-14 10:11:48,660 DEBUG
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
test.slave152.com:8021 sending out status for 0 containers
2013-12-14 10:11:48,687 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
container_1386987056408_0001_02_000001
2013-12-14 10:11:48,688 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: FINISH_APPLICATION sent to absent application
application_1386987056408_0001

Does that mean there are 0 containers on this NM?



On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:

> Hi Vinod,
> I enabled DEBUG log on both RM and NM.
> RM Log:
>
> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Sending sasl message state: NEGOTIATE
>
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Received SASL message state: NEGOTIATE
> auths {
>   method: "TOKEN"
>   mechanism: "DIGEST-MD5"
>   protocol: ""
>   serverId: "default"
>   challenge:
> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
> }
>
> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Get token info proto:interface
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
> 2013-12-14 09:17:47,216 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN]
>
> NM Log:
> 2013-12-14 09:17:47,104 DEBUG
> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
> (auth:SIMPLE)
> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
> Created SASL server with mechanism = DIGEST-MD5
> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
> message state: NEGOTIATE
> auths {
>   method: "TOKEN"
>   mechanism: "DIGEST-MD5"
>   protocol: ""
>   serverId: "default"
>   challenge:
> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
> }
>
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
> Wrote 166 bytes.
> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader
> #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of active
> connections: 0
>
> I am using HA based on trunk, there is in the log as:
> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
> /home/hadoop/dn_socket
> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token found
> for logical URI hdfs://test-cluster
>
> Please take a look, Thanks very much.
>
>
>
> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
>> Thanks Vinod,
>>
>> I am not chang MR AppMaster. I'll enable debug logging then paste here.
>>
>>
>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> Is it stock MR or your modified MR AppMaster?
>>>
>>> Can you enable debug logging on NodeManager as well as the MR
>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>> based auth by default, so you should look if AM has the NMToken or not.
>>> Also check the NM logs.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>
>>> > Hi Dears,
>>> >
>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
>>> now.
>>> >
>>> > I am using trunk in the test unsecure cluster, when submit Job, it
>>> > complained:
>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed on
>>> > local exception: java.io.IOException:
>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>> > at
>>> >
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> > at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message