hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Container authenticate failed after YARN-961
Date Sat, 14 Dec 2013 04:35:25 GMT
HI,


I looked at log carefully, then I found:
2013-12-14 12:29:19,184 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
container_1386995306465_0001_02_000001
2013-12-14 12:29:19,185 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: FINISH_APPLICATION sent to absent application
application_1386995306465_0001

It sounds like AMLauncher request container for  AM, but node manager
cannot get container.

so how to solve it?


On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:

> Maybe My yarn was configured incorrect.
> in the NM log:
> 2013-12-14 10:11:48,660 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
> health-status : true,
> 2013-12-14 10:11:48,660 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> test.slave152.com:8021 sending out status for 0 containers
> 2013-12-14 10:11:48,687 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: KILL_CONTAINER sent to absent container
> container_1386987056408_0001_02_000001
> 2013-12-14 10:11:48,688 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1386987056408_0001
>
> Does that mean there are 0 containers on this NM?
>
>
>
> On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
>> Hi Vinod,
>> I enabled DEBUG log on both RM and NM.
>> RM Log:
>>
>> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Sending sasl message state: NEGOTIATE
>>
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Received SASL message state: NEGOTIATE
>> auths {
>>   method: "TOKEN"
>>   mechanism: "DIGEST-MD5"
>>   protocol: ""
>>   serverId: "default"
>>   challenge:
>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>> }
>>
>> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Get token info proto:interface
>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
>> 2013-12-14 09:17:47,216 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
>> cause:org.apache.hadoop.security.AccessControlException: Client cannot
>> authenticate via:[TOKEN]
>>
>> NM Log:
>> 2013-12-14 09:17:47,104 DEBUG
>> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
>> (auth:SIMPLE)
>> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
>> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
>> Created SASL server with mechanism = DIGEST-MD5
>> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
>> message state: NEGOTIATE
>> auths {
>>   method: "TOKEN"
>>   mechanism: "DIGEST-MD5"
>>   protocol: ""
>>   serverId: "default"
>>   challenge:
>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>> }
>>
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>> Wrote 166 bytes.
>> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader
>> #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of
>> active connections: 0
>>
>> I am using HA based on trunk, there is in the log as:
>> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
>> /home/hadoop/dn_socket
>> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token
>> found for logical URI hdfs://test-cluster
>>
>> Please take a look, Thanks very much.
>>
>>
>>
>> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>
>>> Thanks Vinod,
>>>
>>> I am not chang MR AppMaster. I'll enable debug logging then paste here.
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> Is it stock MR or your modified MR AppMaster?
>>>>
>>>> Can you enable debug logging on NodeManager as well as the MR
>>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>>> based auth by default, so you should look if AM has the NMToken or not.
>>>> Also check the NM logs.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>>
>>>> > Hi Dears,
>>>> >
>>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
>>>> now.
>>>> >
>>>> > I am using trunk in the test unsecure cluster, when submit Job, it
>>>> > complained:
>>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed
>>>> on
>>>> > local exception: java.io.IOException:
>>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>>> > at
>>>> >
>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>>> > at
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> > at
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> > at java.lang.Thread.run(Thread.java:744)
>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message