flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7870) SlotPool should cancel the slot request to RM if not need any more.
Date Fri, 03 Nov 2017 06:33:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237161#comment-16237161

ASF GitHub Bot commented on FLINK-7870:

Github user shuai-xu commented on a diff in the pull request:

    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManager.java
    @@ -302,7 +302,12 @@ public boolean unregisterSlotRequest(AllocationID allocationId) {
     		PendingSlotRequest pendingSlotRequest = pendingSlotRequests.remove(allocationId);
     		if (null != pendingSlotRequest) {
    -			cancelPendingSlotRequest(pendingSlotRequest);
    +			if (pendingSlotRequest.isAssigned()) {
    +				cancelPendingSlotRequest(pendingSlotRequest);
    +			}
    +			else {
    +				resourceActions.cancelResourceAllocation(pendingSlotRequest.getResourceProfile());
    --- End diff --
    Yes, the SlotManager can decide to release the resource more than needed. But in a worst
    1. Now the MESOS or YARN cluster have not enough resource.
    2. A job ask for 100 worker of size A;
    3. As there are not enough resource, the job failover, the previous 100 is not cancelled,
it ask another 100.
    4. This repeated several times, the pending requests for worker of size A reaches 10000.
    5. A worker of size B crashed, so the job now only need 100 woker of size A and 1 worker
of size B. But the YARN or MESOS think the job need 10000 A and 1 B as the request are never
    6. The MESOS/YARN now have some resources for 110 A, more than 100 A and 1 B, and it begin
to assign resource for the job, but it first try to allocate 10000 containers of size A, and
the job still can not be started as it is lack of container B. 
    7. This may cause the job can not be started even when the cluster resource is now enough
in a long time.
    8. And this did happen in our cluster, as our cluster is busy. So I think it's better
to keep this protocol, and different resource managers can treat this protocol according to
their need.

> SlotPool should cancel the slot request to RM if not need any more.
> -------------------------------------------------------------------
>                 Key: FLINK-7870
>                 URL: https://issues.apache.org/jira/browse/FLINK-7870
>             Project: Flink
>          Issue Type: Bug
>          Components: Cluster Management
>            Reporter: shuai.xu
>            Assignee: shuai.xu
>            Priority: Major
>              Labels: flip-6
> 1. SlotPool will request slot to rm if its slots are not enough.
> 2. If a slot request is not fulfilled in a certain time, SlotPool will treat the request
as timeout and send a new slot request by triggering a failover in JobMaster, the previous
request is not needed any more, but rm does not know it.
> 3. This may cause the rm request much more resource than the job really need.
> For example:
> 1. A job need 100 slots. RM request 100 container to YARN.
> 2. But YARN is busy now, it has no resource for the job.
> 3. The job failover as the resource request not fulfilled in time.
> 4. It ask 100 slots again, now RM request 200 container to YARN.
> 5. If failover server time, the containers request  will become more and more.
> 6. Now YARN has resource, it will find that the job may need thousands of containers.
This is a waste of resources.

This message was sent by Atlassian JIRA

View raw message