flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7878) Extend the resource type user can define in ResourceSpec
Date Tue, 07 Nov 2017 07:30:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241620#comment-16241620

ASF GitHub Bot commented on FLINK-7878:

Github user shuai-xu commented on the issue:

    @tillrohrmann, Yes, the Resource is a little too generic and prone to typos. However,
the resource are various and closely related to the platform(YARN/MESOS), only a GPUResource
and FPGAResource may not satisfy user's need. For example, we have at lease two types of FPGA
resources in our cluster. And it could consider the users who need to specify extended resources
as advanced users. General users only need to know vcore and memory, which are already defined
in ResurceSpec. Advanced users should be familiar with not only flink but also the resource
platform. They should know the resources types YARN/MESOS supports. And, If flink resource
manager passes all the extended resource to YARN/MESOS when starting a container, it need
not change when adding a new resource type only if YARN/MESOS can recognize it from extended
resources. There has to be a compromise between extendibility and ease of use. I suggest we
can add a general GPUResource and FPGAResource for general use while still keeping the Resource
for extension. Does this make sense?

> Extend the resource type user can define in ResourceSpec
> --------------------------------------------------------
>                 Key: FLINK-7878
>                 URL: https://issues.apache.org/jira/browse/FLINK-7878
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataSet API, DataStream API
>            Reporter: shuai.xu
>            Assignee: shuai.xu
>              Labels: flip-6
> Now, flink only support user define how much CPU and MEM used in an operator, but now
the resource in a cluster is various. For example, an application for image processing may
need GPU, some others may need FPGA. 
> Only CPU and MEM is not enough, and the resource type is becoming more and more, so we
need to make the ResourSpec extendible.

This message was sent by Atlassian JIRA

View raw message