flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Wang (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-9953) Active Kubernetes integration
Date Mon, 21 Oct 2019 05:05:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955721#comment-16955721
] 

Yang Wang edited comment on FLINK-9953 at 10/21/19 5:04 AM:
------------------------------------------------------------

I do not insist to introduce the `kubernetes-job.sh`. If we could make the `flink run` more
flexible, it is reasonable for me.

1. No jar is specified

In this way, before the jm/tm pod, the jar has already in the image. The job graph is generated
in the jm pod. Since the job is submitted on the jm side, we could only support detach mode.
{code:java}
./bin/flink run -d -m kubernetes-cluster -kid flink-perjob-example-2 \
-ki flink-k8s-windowjoin:latest \
-kjob-class org.apache.flink.streaming.examples.join.WindowJoin \
-kuser-jar /path/in/the/image/WinowJoin.jar
{code}
2. Client local jar is specified

We need to deploy a session cluster to simulate per job mode. Since the jm need to start first,
and then Flink client submit the user jar and job graph to jm pod through rest. Detach and
attach mode are both supported.
{code:java}
./bin/flink run [-d] -m kubernetes-cluster -kid flink-perjob-example-2 \
-ki flink-k8s:latest \
-kjob-class org.apache.flink.streaming.examples.join.WindowJoin \
/path/in/the/client/WinowJoin.jar
{code}
The primary reason to introduce `kubernetes-job.sh` is for #1. And `flink run` is always going
to #2.

[~trohrmann] [~felixzheng] , waiting for your response.


was (Author: fly_in_gis):
[~felixzheng]

I do not insist to introduce the `kubernetes-job.sh`. If we could make the `flink run` more
flexible, it is reasonable for me.

1. No jar is specified

In this way, before the jm/tm pod, the jar has already in the image. The job graph is generated
in the jm pod. Since the job is submitted on the jm side, we could only support detach mode.
{code:java}
./bin/flink run -d -m kubernetes-cluster -kid flink-perjob-example-2 \
-ki flink-k8s-windowjoin:latest \
-kjob-class org.apache.flink.streaming.examples.join.WindowJoin \
-kuser-jar /path/in/the/image/WinowJoin.jar
{code}
2. Client local jar is specified

We need to deploy a session cluster to simulate per job mode. Since the jm need to start first,
and then Flink client submit the user jar and job graph to jm pod through rest. Detach and
attach mode are both supported.
{code:java}
./bin/flink run [-d] -m kubernetes-cluster -kid flink-perjob-example-2 \
-ki flink-k8s:latest \
-kjob-class org.apache.flink.streaming.examples.join.WindowJoin \
/path/in/the/client/WinowJoin.jar
{code}
The primary reason to introduce `kubernetes-job.sh` is for #1. And `flink run` is always going
to #2.

[~trohrmann], waiting for your response.

> Active Kubernetes integration
> -----------------------------
>
>                 Key: FLINK-9953
>                 URL: https://issues.apache.org/jira/browse/FLINK-9953
>             Project: Flink
>          Issue Type: New Feature
>          Components: Runtime / Coordination
>            Reporter: Till Rohrmann
>            Assignee: Yang Wang
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is the umbrella issue tracking Flink's active Kubernetes integration. Active means
in this context that the {{ResourceManager}} can talk to Kubernetes to launch new pods similar
to Flink's Yarn and Mesos integration.
> Phase1 implementation will have complete functions to make flink running on kubernetes.
Phrase2 is mainly focused on production optimization, including k8s native high-availability,
storage, network, log collector and etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message