beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5455) Don't info log for every bundle in the python sdk
Date Fri, 21 Sep 2018 01:26:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5455?focusedWorklogId=146186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146186
]

ASF GitHub Bot logged work on BEAM-5455:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 21/Sep/18 01:25
            Start Date: 21/Sep/18 01:25
    Worklog Time Spent: 10m 
      Work Description: mwylde opened a new pull request #6459: [BEAM-5455] Log per-bundle
messages at debug in python sdk
URL: https://github.com/apache/beam/pull/6459
 
 
   This PR changes several log statements in the python SDK from info to debug. Without this
change, the sdk can log at info level 10+ messages per bundle. For the portable flink runtime
this means many logs per message consumed by the pipeline.
   
   ------------------------
   
   Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically
link the pull request to the issue.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`)
to look at it.
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
   Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 146186)
            Time Spent: 10m
    Remaining Estimate: 0h

> Don't info log for every bundle in the python sdk
> -------------------------------------------------
>
>                 Key: BEAM-5455
>                 URL: https://issues.apache.org/jira/browse/BEAM-5455
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-harness
>            Reporter: Micah Wylde
>            Assignee: Robert Bradshaw
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every bundle. For
the portable flink runner, which uses a bundle size of 1 in streaming mode, this produces
huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DataOutputOperation
>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation
window output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation
pair_with_one output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation
timestamp output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation
decode output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DataInputOperation
receivers=[ConsumerSet[.out0, coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DataInputOperation
receivers=[ConsumerSet[.out0, coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation
decode output_tags=['out'], receivers=[ConsumerSet[decode.out0, coder=WindowedValueCoder[FastPrimitivesCoder],
len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation
timestamp output_tags=['out'], receivers=[ConsumerSet[timestamp.out0, coder=WindowedValueCoder[FastPrimitivesCoder],
len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation
pair_with_one output_tags=['out'], receivers=[ConsumerSet[pair_with_one.out0, coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder,
FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation
window output_tags=['out'], receivers=[ConsumerSet[window.out0, coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder,
FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DataOutputOperation
>
> {code}
>  
> These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message