hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Container authenticate failed after YARN-961
Date Sun, 15 Dec 2013 12:38:49 GMT
Resolved. It's my fault, I changed some code from the trunk, which leads to
Token cannot generated.


On Sat, Dec 14, 2013 at 12:35 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:

> HI,
>
>
> I looked at log carefully, then I found:
> 2013-12-14 12:29:19,184 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: KILL_CONTAINER sent to absent container
> container_1386995306465_0001_02_000001
> 2013-12-14 12:29:19,185 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1386995306465_0001
>
> It sounds like AMLauncher request container for  AM, but node manager
> cannot get container.
>
> so how to solve it?
>
>
> On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
>> Maybe My yarn was configured incorrect.
>> in the NM log:
>> 2013-12-14 10:11:48,660 DEBUG
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
>> health-status : true,
>> 2013-12-14 10:11:48,660 DEBUG
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> test.slave152.com:8021 sending out status for 0 containers
>> 2013-12-14 10:11:48,687 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Event EventType: KILL_CONTAINER sent to absent container
>> container_1386987056408_0001_02_000001
>> 2013-12-14 10:11:48,688 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Event EventType: FINISH_APPLICATION sent to absent application
>> application_1386987056408_0001
>>
>> Does that mean there are 0 containers on this NM?
>>
>>
>>
>> On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>
>>> Hi Vinod,
>>> I enabled DEBUG log on both RM and NM.
>>> RM Log:
>>>
>>> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Sending sasl message state: NEGOTIATE
>>>
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Received SASL message state: NEGOTIATE
>>> auths {
>>>   method: "TOKEN"
>>>   mechanism: "DIGEST-MD5"
>>>   protocol: ""
>>>   serverId: "default"
>>>   challenge:
>>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>>> }
>>>
>>> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Get token info proto:interface
>>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
>>> 2013-12-14 09:17:47,216 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
>>> cause:org.apache.hadoop.security.AccessControlException: Client cannot
>>> authenticate via:[TOKEN]
>>>
>>> NM Log:
>>> 2013-12-14 09:17:47,104 DEBUG
>>> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
>>> (auth:SIMPLE)
>>> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
>>> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
>>> Created SASL server with mechanism = DIGEST-MD5
>>> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
>>> message state: NEGOTIATE
>>> auths {
>>>   method: "TOKEN"
>>>   mechanism: "DIGEST-MD5"
>>>   protocol: ""
>>>   serverId: "default"
>>>   challenge:
>>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>>> }
>>>
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>>> Wrote 166 bytes.
>>> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket
>>> Reader #1 for port 8021: disconnecting client 10.2.8.148:34175. Number
>>> of active connections: 0
>>>
>>> I am using HA based on trunk, there is in the log as:
>>> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
>>> /home/hadoop/dn_socket
>>> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token
>>> found for logical URI hdfs://test-cluster
>>>
>>> Please take a look, Thanks very much.
>>>
>>>
>>>
>>> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>
>>>> Thanks Vinod,
>>>>
>>>> I am not chang MR AppMaster. I'll enable debug logging then paste
>>>> here.
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> Is it stock MR or your modified MR AppMaster?
>>>>>
>>>>> Can you enable debug logging on NodeManager as well as the MR
>>>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>>>> based auth by default, so you should look if AM has the NMToken or not.
>>>>> Also check the NM logs.
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>>>
>>>>> > Hi Dears,
>>>>> >
>>>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce
>>>>> Job now.
>>>>> >
>>>>> > I am using trunk in the test unsecure cluster, when submit Job,
it
>>>>> > complained:
>>>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed
>>>>> on
>>>>> > local exception: java.io.IOException:
>>>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>>>> > at
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> > at
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> > at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message