beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
Date Tue, 18 Sep 2018 14:43:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-3089?focusedWorklogId=145308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145308
]

ASF GitHub Bot logged work on BEAM-3089:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 18/Sep/18 14:42
            Start Date: 18/Sep/18 14:42
    Worklog Time Spent: 10m 
      Work Description: mxm opened a new pull request #6426: [BEAM-3089] Fix default values
in FlinkPipelineOptions / Add tests
URL: https://github.com/apache/beam/pull/6426
 
 
   
   - [BEAM-3089] Use Flink cluster parallelism if no parallelism provided
   
   The Runner always defaulted to 1, even if the Flink cluster had a default
   parallelism set. With this patch, when no parallelism has been provided, the
   cluster default will be used.
   
   This was working before, it is a regression of
   cdd2544.
   
   - [BEAM-3089] Add test for FlinkExecutionEnvironments
   - Revert default checkpointing mode to EXACTLY_ONCE
   - Use default for checkpoint timeout
   - [BEAM-3089] Test default values of FlinkPipelineOptions
   - Set default master url to [auto] 
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
   Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 145308)
    Time Spent: 1.5h  (was: 1h 20m)

> Issue with setting the parallelism at client level using Flink runner
> ---------------------------------------------------------------------
>
>                 Key: BEAM-3089
>                 URL: https://issues.apache.org/jira/browse/BEAM-3089
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.0.0
>         Environment: I am using Flink 1.2.1 running on Docker, with Task Managers distributed
across different VMs as part of a Docker Swarm.
>            Reporter: Thalita Vergilio
>            Assignee: Grzegorz KoĊ‚akowski
>            Priority: Major
>              Labels: docker, flink, parallel-deployment
>             Fix For: 2.8.0
>
>         Attachments: flink-ui-parallelism.png
>
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When uploading an Apache Beam application using the Flink Web UI, the parallelism set
at job submission doesn't get picked up. The same happens when submitting a job using the
Flink CLI.
> In both cases, the parallelism ends up defaulting to 1.
> When I set the parallelism programmatically within the Apache Beam code, it works: {{flinkPipelineOptions.setParallelism(4);}}
> I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory
class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values
passed to Flink, then defaults to 1 if it doesn't find anything.
> Any ideas on how this could be fixed or worked around? I need to be able to change the
parallelism dynamically, so the programmatic approach won't really work for me, nor will setting
the Flink configuration at system level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message