beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kamil Szewczyk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner
Date Tue, 28 Nov 2017 23:46:00 GMT
Kamil Szewczyk created BEAM-3268:
------------------------------------

             Summary: getPerDestinationOutputFilenames() is getting processed before write
is finished on dataflow runner
                 Key: BEAM-3268
                 URL: https://issues.apache.org/jira/browse/BEAM-3268
             Project: Beam
          Issue Type: Bug
          Components: runner-dataflow
    Affects Versions: 2.3.0
            Reporter: Kamil Szewczyk
            Assignee: Thomas Groh


While running filebased-io-test we found dataflow-runnner misbehaving. We run tests using
single pipeline and without using Reshuffling between writing and reading dataflow jobs are
unsuccessful because the runner tries to access the files that were not created yet. 

On the picture the difference between execution of writting is presented. On the left there
is working example with Reshuffling added and on the right without it.
!comparison.png|thumbnail!

Steps to reproduce: substitute your-bucket-name wit your valid bucket.

{code:java}
mvn -e -Pio-it verify -pl sdks/java/io/file-based-io-tests -DintegrationTestPipelineOptions='["--runner=dataflow",
"--filenamePrefix=gs://your-bucket-name/TEXTIO_IT"]' -Pdataflow-runner
{code}

Then look on the cloud console and job should fail.
Now add Reshuffling to sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
as in the example.
{code:java}
.getPerDestinationOutputFilenames().apply(Values.<String>create())
        .apply(Reshuffle.<String>viaRandomKey());

    PCollection<String> consolidatedHashcode = testFilenames
{code}

and trigger previously used maven command to see it working in the console right now.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message