beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3310) Push metrics to a backend in an runner agnostic way
Date Fri, 26 Jan 2018 22:20:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341683#comment-16341683
] 

Luke Cwik commented on BEAM-3310:
---------------------------------

 

This could work as an extension to a pipeline if metrics sinks were defined part of pipeline
construction and then metrics events would be sent to SDKs either on the control stream or
over the data plane.

 

Alternatively this could be baked inside an SDK harness but would be limited to only reporting
metrics for that given SDK instead of across all SDKs.

> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
>                 Key: BEAM-3310
>                 URL: https://issues.apache.org/jira/browse/BEAM-3310
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics (either
at the end of the pipeline or while it runs) because they don't have all the same capabilities
towards metrics (e.g. spark runner configures sinks  like csv, graphite or in memory sinks
using the spark engine conf). The target is to push the metrics in the common runner code
so that no matter the chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message