hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Teke (Jira)" <j...@apache.org>
Subject [jira] [Created] (YARN-10850) TimelineService v2 lists containers for all attempts when filtering for one
Date Wed, 07 Jul 2021 14:41:00 GMT
Benjamin Teke created YARN-10850:
------------------------------------

             Summary: TimelineService v2 lists containers for all attempts when filtering
for one
                 Key: YARN-10850
                 URL: https://issues.apache.org/jira/browse/YARN-10850
             Project: Hadoop YARN
          Issue Type: Bug
          Components: timelinereader
            Reporter: Benjamin Teke


When using the command

{code:java}
yarn container -list <app_attempt_id>
{code}

with an application attempt ID based on the help only the containers for that attempt should
be listed.

{code:java}
-list <Application Name or Attempt ID>    List containers for application
                                          attempt when application
                                          attempt ID is provided. When
                                          application name is provided,
                                          then it finds the instances of
                                          the application based on app's
                                          own implementation, and
                                          -appTypes option must be
                                          specified unless it is the
                                          default yarn-service type. With
                                          app name, it supports optional
                                          use of -version to filter
                                          instances based on app version,
                                          -components to filter instances
                                          based on component names,
                                          -states to filter instances
                                          based on instance state.
{code}

When TimelineService v2 is enabled all of the containers for the application are returned.


{code:java}
hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn applicationattempt -list
application_1625124233002_0007
21/07/01 09:32:23 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/,
clusterId=yarn-cluster
21/07/01 09:32:24 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:24 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
Total number of application attempts :2
         ApplicationAttempt-Id	               State	                    AM-Container-Id	 
                     Tracking-URL
appattempt_1625124233002_0007_000001	              FAILED	container_e43_1625124233002_0007_01_000001
http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
appattempt_1625124233002_0007_000002	              KILLED	container_e43_1625124233002_0007_02_000001
http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
{code}

Querying the 2 app attempts produces the same output:

{code:java}
hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000001
21/07/01 09:32:35 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/,
clusterId=yarn-cluster
21/07/01 09:32:35 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
21/07/01 09:32:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
Total number of containers :12
                  Container-Id	          Start Time	         Finish Time	               State
                Host	   Node Http Address	                            LOG-URL
container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	    
           null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
{code}
{code:java}
hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000002
21/07/01 09:32:50 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/,
clusterId=yarn-cluster
21/07/01 09:32:51 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
21/07/01 09:32:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
21/07/01 09:32:51 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
Total number of containers :12
                  Container-Id	          Start Time	         Finish Time	               State
                Host	   Node Http Address	                            LOG-URL
container_e43_1625124233002_0007_02_000004	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
container_e43_1625124233002_0007_02_000005	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
container_e43_1625124233002_0007_02_000003	                 N/A	                 N/A	    
           null	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
container_e43_1625124233002_0007_02_000002	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
container_e43_1625124233002_0007_02_000006	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
container_e43_1625124233002_0007_02_000001	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
container_e43_1625124233002_0007_01_000005	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000002.hwx.site:25454	ctr-e172-1620330694487-146061-01-000002.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
container_e43_1625124233002_0007_01_000004	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
container_e43_1625124233002_0007_01_000002	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000005.hwx.site:25454	ctr-e172-1620330694487-146061-01-000005.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
container_e43_1625124233002_0007_01_000006	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000006.hwx.site:25454	ctr-e172-1620330694487-146061-01-000006.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
container_e43_1625124233002_0007_01_000003	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000007.hwx.site:25454	ctr-e172-1620330694487-146061-01-000007.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
container_e43_1625124233002_0007_01_000001	                 N/A	                 N/A	    
       COMPLETE	ctr-e172-1620330694487-146061-01-000004.hwx.site:25454	ctr-e172-1620330694487-146061-01-000004.hwx.site:8042
http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
{code}

The reason for this is that the [container collector method|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java#L135]
in TimelineServiceV2 gets the *application ID* from the *attempt ID* and collects the [containers
for the application|https://github.com/apache/hadoop/blob/a3edfddcf7822ea13bdf4858672eb82cea5e0b5f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java#L116].

The other implementations however are using the application attempt ID: [ApplicationHistoryManagerOnTimelineStore|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java#L221]
and [ApplicationHistoryManagerImpl|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java#L235].

TSv2 should behave similarly to v1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message