flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2084) Create a dedicated streaming mode
Date Fri, 22 May 2015 15:23:17 GMT
Stephan Ewen created FLINK-2084:

             Summary: Create a dedicated streaming mode
                 Key: FLINK-2084
                 URL: https://issues.apache.org/jira/browse/FLINK-2084
             Project: Flink
          Issue Type: Improvement
          Components: Distributed Runtime
    Affects Versions: 0.9
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen
             Fix For: 0.9

As per discussion on the mailing list


 - We add a dedicated streaming mode for now. The streaming mode supersedes the batch mode,
so it can run both type of programs.

 - The streaming mode sets the memory manager to "lazy allocation".
    -> So long as it runs pure streaming jobs, the full heap will be
available to window buffers and UDFs.
    -> Batch programs can still run, so mixed workloads are not prevented.
Batch programs are a bit less robust there, because the memory manager does not pre-allocate
memory. UDFs can eat into Flink's memory portion.

 - The streaming mode starts the necessary configured components/services for state backups

Over the next versions, we want to bring these things together:
  - use the managed memory for window buffers
  - on-demand starting of the state backend

Then, we deprecate the streaming mode, let both modes start the cluster in the same way.

This message was sent by Atlassian JIRA

View raw message