beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-5520) Flink runner per operator SDK harness option
Date Fri, 28 Sep 2018 05:03:00 GMT
Thomas Weise created BEAM-5520:
----------------------------------

             Summary: Flink runner per operator SDK harness option
                 Key: BEAM-5520
                 URL: https://issues.apache.org/jira/browse/BEAM-5520
             Project: Beam
          Issue Type: Improvement
          Components: runner-flink
            Reporter: Thomas Weise
            Assignee: Thomas Weise


For streaming pipelines, the runner currently uses a single SDK harness process for all subtasks
of a job that get deployed on the same task manager. In common deployments with 16 or more
tasks slots, many executable stage operators all use the same SDK harness process. To scale,
we need an option to run separate harness processes per subtask.

   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message