beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5412) TFRecordIO fails with records larger than 8K
Date Thu, 20 Sep 2018 00:07:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5412?focusedWorklogId=145878&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145878
]

ASF GitHub Bot logged work on BEAM-5412:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 20/Sep/18 00:06
            Start Date: 20/Sep/18 00:06
    Worklog Time Spent: 10m 
      Work Description: rangadi commented on a change in pull request #6440: [BEAM-5412][BEAM-5408]
Fixes a bug that limited the size of TFRecords
URL: https://github.com/apache/beam/pull/6440#discussion_r219000921
 
 

 ##########
 File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
 ##########
 @@ -287,6 +288,18 @@ public void runTestRoundTripZlibFilesWithAuto() throws IOException {
     runTestRoundTrip(LARGE, 10, ".tfrecords", DEFLATE, AUTO);
   }
 
+  @Test
+  @Category(NeedsRunner.class)
+  public void runTestRoundTripLargeRecords() throws IOException {
 
 Review comment:
   Do both of the new tests fail or just the GZIP version without this fix? (just curious).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 145878)
    Time Spent: 40m  (was: 0.5h)

> TFRecordIO fails with records larger than 8K
> --------------------------------------------
>
>                 Key: BEAM-5412
>                 URL: https://issues.apache.org/jira/browse/BEAM-5412
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-text
>    Affects Versions: 2.4.0
>            Reporter: Raghu Angadi
>            Assignee: Chamikara Jayalath
>            Priority: Major
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> This was reported on [Stackoverflow|https://stackoverflow.com/questions/52284639/beam-java-sdk-with-tfrecord-and-compression-gzip].
TFRecordIO reader assumes a single call to {{channel.read()}} returns as much as can fit in
the input buffer. {{read()}} can return fewer bytes than requested. Assert failure : https://github.com/apache/beam/blob/release-2.4.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L642



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message