flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2084) Create a dedicated streaming mode
Date Mon, 25 May 2015 12:35:17 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558202#comment-14558202

ASF GitHub Bot commented on FLINK-2084:

Github user rmetzger commented on a diff in the pull request:

    --- Diff: flink-dist/src/main/flink-bin/bin/jobmanager.sh ---
    @@ -80,7 +81,7 @@ case $STARTSTOP in
             rotateLogFile $out
             echo "Starting Job Manager"
    -        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList
"$FLINK_JM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.runtime.jobmanager.JobManager
--executionMode $EXECUTIONMODE --configDir "$FLINK_CONF_DIR"  > "$out" 2>&1 <
/dev/null &
    +        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList
"$FLINK_JM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.runtime.jobmanager.JobManager
--configDir "$FLINK_CONF_DIR" --executionMode $EXECUTIONMODE --streamingMode "$STREAMINGMODE"
> "$out" 2>&1 < /dev/null &
    --- End diff --
    With this change, the script now expects three arguments, right?
    If you, you need to update this line as well
    echo "Please specify 'start (cluster|local)' or stop"

> Create a dedicated streaming mode
> ---------------------------------
>                 Key: FLINK-2084
>                 URL: https://issues.apache.org/jira/browse/FLINK-2084
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>    Affects Versions: 0.9
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.9
> As per discussion on the mailing list
> http://mail-archives.apache.org/mod_mbox/flink-dev/201505.mbox/browser
>  - We add a dedicated streaming mode for now. The streaming mode supersedes the batch
mode, so it can run both type of programs.
>  - The streaming mode sets the memory manager to "lazy allocation".
>     -> So long as it runs pure streaming jobs, the full heap will be
> available to window buffers and UDFs.
>     -> Batch programs can still run, so mixed workloads are not prevented.
> Batch programs are a bit less robust there, because the memory manager does not pre-allocate
memory. UDFs can eat into Flink's memory portion.
>  - The streaming mode starts the necessary configured components/services for state backups
> Over the next versions, we want to bring these things together:
>   - use the managed memory for window buffers
>   - on-demand starting of the state backend
> Then, we deprecate the streaming mode, let both modes start the cluster in the same way.

This message was sent by Atlassian JIRA

View raw message