flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2084) Create a dedicated streaming mode
Date Fri, 22 May 2015 16:57:18 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556435#comment-14556435

ASF GitHub Bot commented on FLINK-2084:

GitHub user StephanEwen opened a pull request:


    [FLINK-2084] [core] Add an option to start Flink in streaming mode

    The streaming mode subsumes the batch mode, as it allows to run both kinds of programs.
Batch programs will most likely run slightly less efficient than in pure batch mode, due to
lazy memory allocation (see below)
    Adds new startup options to the JobManager and TaskManager.
    Adds additional scripts `bin/start-cluster-streaming.sh` and `start-local-streaming.sh`.
    Currently, streaming mode only sets the memory manager to lazy memory allocation to ensure
heap is not blocked by batch memory manager
    Includes [FLINK-2085] To support on-demand memory allocation in the memory manager.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StephanEwen/incubator-flink streaming_mode

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #718
commit 7b58e2d49521e955ddacfa5586659df01fe57c57
Author: Stephan Ewen <sewen@apache.org>
Date:   2015-05-22T16:35:40Z

    [FLINK-2085] [runtime] Add an option to let the MemoryManager allocate and release memory
as needed.
    This is an alternative mode to the current mode that pre-allocates all memory.
    The default remains to pre-allocate all memory.

commit 43978cd979114858ad2f200f04bd1723693c2c3e
Author: Stephan Ewen <sewen@apache.org>
Date:   2015-05-22T15:12:45Z

    [FLINK-2084] [core] Add an option to start Flink in streaming mode
     - Streaming mode sets the memory manager to lazy memory allocation to ensure
       heap is not blocked by batch memory manager

commit 1df1f12e367b4b89b56e9e47dbe5a7a11e5e1583
Author: Stephan Ewen <sewen@apache.org>
Date:   2015-05-22T15:24:19Z

    [tests] Adjust tests for dedicated streaming mode and clean up test bases.


> Create a dedicated streaming mode
> ---------------------------------
>                 Key: FLINK-2084
>                 URL: https://issues.apache.org/jira/browse/FLINK-2084
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>    Affects Versions: 0.9
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.9
> As per discussion on the mailing list
> http://mail-archives.apache.org/mod_mbox/flink-dev/201505.mbox/browser
>  - We add a dedicated streaming mode for now. The streaming mode supersedes the batch
mode, so it can run both type of programs.
>  - The streaming mode sets the memory manager to "lazy allocation".
>     -> So long as it runs pure streaming jobs, the full heap will be
> available to window buffers and UDFs.
>     -> Batch programs can still run, so mixed workloads are not prevented.
> Batch programs are a bit less robust there, because the memory manager does not pre-allocate
memory. UDFs can eat into Flink's memory portion.
>  - The streaming mode starts the necessary configured components/services for state backups
> Over the next versions, we want to bring these things together:
>   - use the managed memory for window buffers
>   - on-demand starting of the state backend
> Then, we deprecate the streaming mode, let both modes start the cluster in the same way.

This message was sent by Atlassian JIRA

View raw message