flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zentol <...@git.apache.org>
Subject [GitHub] flink pull request #3106: [FLINK-????] FIx inconsistent numBytesIn/Out metri...
Date Thu, 12 Jan 2017 14:08:58 GMT
GitHub user zentol opened a pull request:

    https://github.com/apache/flink/pull/3106

    [FLINK-????] FIx inconsistent numBytesIn/Out metrics

    This PR fixes _some_ jira issue (i can't find the correct issue ID right now since JIRA
is being stupid) regarding inconsistent numBytesIn/Out metrics.
    
    Given 2 subsequent tasks A->B the numBytesOut count of A was lower than the numBytesIn(Local/Remote)
count of B by a huge margin, although they should be (nearly) identical.
    
    The problem is that A was counting how much bytes the serialized records were using, whereas
B was counting how large the ```Buffer```s were that it received. A ```Buffer``` contains
the following data:
    ```
    |size_of_R1|R1|size_of_R2|R2|...
    ```
    where as R1/R2 are serialized records and the sizes denote the serialized length of their
respective record.
    
    So, while A was adding ```sizeOf(R1)```, B was adding sizeOf(size_of_R1) + sizeOf(R1).
    
    A was simply not accounting for the added bytes that size_of_RX were using, which is what
this PR is fixing by adding ```4``` after the serialization of each record.
    
    cc @uce 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zentol/flink _metrics_incon

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3106.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3106
    
----
commit 5300a35ac6ee8815d07a739c9b370ae475f0d9a0
Author: zentol <chesnay@apache.org>
Date:   2017-01-12T13:59:22Z

    [FLINK-????] FIx inconsistent numBytesIn/Out metrics

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message