flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Moiz S Jinia <moiz.ji...@gmail.com>
Subject Re: Programmatic management of Flink jobs
Date Tue, 09 May 2017 06:23:14 GMT
Thanks. Do you foresee any problems with canceling/starting the jobs
whenever the patterns change?

Moiz

On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi,
> Yes, this would work even though it requires canceling/restarting the job
> whenever the patterns change.
>
> Best,
> Aljoscha
>
> On 3. May 2017, at 12:36, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
>
> The kind of program I intend to submit would be one that sets up
> a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and
> uses a PatternStream over the kafka events. I could have the jar for this
> program readily available in the "jobmanager.web.upload.dir", and use the
> REST API for submitting a program with some configuration params.
>
> Does that sound like it'd work or am I missing something?
>
> Moiz
>
> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
>
>> Not sure I understand Operators. What I need is to have a Pattern that
>> starts consuming from a Kafka stream. And I need the Patterns to come and
>> go.
>>
>> Another option that comes to mind is this -
>> The Patterns I'll need are well known in advance. Only certain parameters
>> such as the time duration of the within clause, and maybe certain filter
>> confitions of the where clause need tweaking. So I could pre-deploy the
>> Patterns (or jobs) and start or stop them (with parameters).
>>
>> Does that sound feasible?
>>
>> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljoscha@apache.org>
>> wrote:
>>
>>> What would the pattern be added to. An existing custom operator?
>>>
>>> The REST interface only allows for managing the lifecycle of a job, not
>>> modifying their graph structure.
>>>
>>> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
>>>
>>> Thanks for the references. Looking at the REST API, would adding new
>>> Patterns not work via this?
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>> monitoring/rest_api.html#submitting-programs
>>>
>>> Moiz
>>>
>>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljoscha@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> For managing a Job you can either use the bin/flink command-line tool
>>>> or the Rest API [1]. As for dynamically adding patterns, that’s outside
of
>>>> the scope of Flink right now. There are, however, some users that
>>>> implemented this on top of Flink, see for example RBEA [2]. The basic idea
>>>> is to use a ConnectedStream where one input is the main input and the other
>>>> input is a control stream that updates the existing patterns.
>>>>
>>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>>>> 1.3/monitoring/rest_api.html
>>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>>>
>>>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com> wrote:
>>>>
>>>> Is there an API that allows remotely adding, modifying, and cancelling
>>>> Flink jobs? Example - changing the time window of a deployed Pattern,
>>>> adding new Patterns, etc.
>>>>
>>>> Whats the best way to go about this? To the end user the Pattern would
>>>> manifest as rules that can be updated anytime.
>>>>
>>>>
>>>> Moiz
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>

Mime
View raw message