See ------------------------------------------ [...truncated 5.63 MB...] Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.395Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.443Z: Unzipping flatten s20-u67-u74 for input s23-reify-value27-c72 Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.478Z: Fusing unzipped copy of WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write, through flatten s20-u67-u74, into producer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.528Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.580Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.621Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.682Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.731Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.777Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.824Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.864Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.900Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.935Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:44.985Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.034Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.085Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.131Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.177Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.232Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.280Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.329Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.376Z: Fusing consumer TextIO.Read/Read/ParDo(SplitBoundedSource) into TextIO.Read/Read/Impulse Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.425Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into TextIO.Read/Read/Reshuffle.ViaRandomKey/Pair with random key Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.471Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.518Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/GroupByWindow into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.569Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Values/Values/Map into TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.616Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.657Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.698Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.742Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.783Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.833Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.880Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into MapElements/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.927Z: Fusing consumer Window.Into()/Window.Assign into ParDo(AddTimestamp) Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:45.970Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.016Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.064Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.110Z: Fusing consumer TextIO.Read/Read/ParDo(ReadFromBoundedSource) into TextIO.Read/Read/Reshuffle.ViaRandomKey/Values/Values/Map Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.160Z: Fusing consumer ParDo(AddTimestamp) into TextIO.Read/Read/ParDo(ReadFromBoundedSource) Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.213Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.255Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.298Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/GroupByWindow Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.342Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.382Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into Window.Into()/Window.Assign Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.435Z: Fusing consumer WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.486Z: Fusing consumer TextIO.Read/Read/Reshuffle.ViaRandomKey/Pair with random key into TextIO.Read/Read/ParDo(SplitBoundedSource) Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:46.529Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.201Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.260Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.307Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Create Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.318Z: Starting 1 workers in us-central1-b... Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.365Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.420Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jan 03, 2019 6:09:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.480Z: Executing operation TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jan 03, 2019 6:09:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:47.929Z: Executing operation TextIO.Read/Read/Impulse+TextIO.Read/Read/ParDo(SplitBoundedSource)+TextIO.Read/Read/Reshuffle.ViaRandomKey/Pair with random key+TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jan 03, 2019 6:09:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:09:58.163Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Jan 03, 2019 6:10:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:10:26.602Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Jan 03, 2019 6:10:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:10:26.655Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jan 03, 2019 6:10:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:10:51.214Z: Workers have started successfully. Jan 03, 2019 6:10:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:10:51.264Z: Workers have started successfully. Jan 03, 2019 6:11:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:12.889Z: Executing operation TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jan 03, 2019 6:11:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:13.028Z: Executing operation TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+TextIO.Read/Read/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+TextIO.Read/Read/Reshuffle.ViaRandomKey/Values/Values/Map+TextIO.Read/Read/ParDo(ReadFromBoundedSource)+ParDo(AddTimestamp)+Window.Into()/Window.Assign+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Jan 03, 2019 6:11:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:21.710Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Jan 03, 2019 6:11:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:21.835Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/GroupByWindow+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Jan 03, 2019 6:11:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:37.127Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Jan 03, 2019 6:11:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:37.242Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write Jan 03, 2019 6:11:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:38.643Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Close Jan 03, 2019 6:11:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:38.893Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jan 03, 2019 6:11:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:44.968Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jan 03, 2019 6:11:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:45.101Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jan 03, 2019 6:11:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:54.111Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jan 03, 2019 6:11:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:54.233Z: Executing operation WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Jan 03, 2019 6:11:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:58.449Z: Cleaning up. Jan 03, 2019 6:12:01 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:11:58.616Z: Stopping worker pool... Jan 03, 2019 6:14:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:14:06.307Z: Autoscaling: Resized worker pool from 1 to 0. Jan 03, 2019 6:14:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:14:06.345Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jan 03, 2019 6:14:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T06:14:06.409Z: Worker pool stopped. Jan 03, 2019 6:14:15 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2019-01-02_22_09_36-4348266898790107185 finished with status DONE. Jan 03, 2019 6:14:15 AM org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess INFO: Success result for Dataflow job 2019-01-02_22_09_36-4348266898790107185. Found 0 success, 0 failures out of 0 expected assertions. Jan 03, 2019 6:14:18 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2019-01-02_22_09_36-4348266898790107185 finished with status DONE. Gradle Test Executor 1 finished executing tests. > Task :beam-runners-google-cloud-dataflow-java-examples:preCommitFnApiWorker org.apache.beam.examples.WordCountIT > testE2EWordCount STANDARD_ERROR Jan 03, 2019 7:03:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2019-01-03T07:03:45.084Z: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at https://cloud.google.com/dataflow/support. Jan 03, 2019 7:03:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T07:03:45.224Z: Cancel request is committed for workflow job: 2019-01-02_22_03_33-18400219030404423797. Jan 03, 2019 7:03:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T07:03:45.323Z: Cleaning up. Jan 03, 2019 7:03:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T07:03:45.417Z: Stopping worker pool... Jan 03, 2019 7:03:46 AM org.apache.beam.runners.dataflow.TestDataflowRunner$ErrorMonitorMessagesHandler process INFO: Dataflow job 2019-01-02_22_03_33-18400219030404423797 threw exception. Failure message was: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at https://cloud.google.com/dataflow/support. Jan 03, 2019 7:06:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T07:06:02.614Z: Autoscaling: Reduced the number of workers to 0 based on the rate of progress in the currently running step(s). Jan 03, 2019 7:06:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-01-03T07:06:02.658Z: Worker pool stopped. Jan 03, 2019 7:06:08 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2019-01-02_22_03_33-18400219030404423797 failed with status FAILED. Jan 03, 2019 7:06:08 AM org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess INFO: Success result for Dataflow job 2019-01-02_22_03_33-18400219030404423797. Found 0 success, 0 failures out of 0 expected assertions. Gradle Test Executor 2 finished executing tests. > Task :beam-runners-google-cloud-dataflow-java-examples:preCommitFnApiWorker FAILED org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED java.lang.RuntimeException: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. You can get help with Cloud Dataflow at https://cloud.google.com/dataflow/support. at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:134) at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90) at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299) at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at org.apache.beam.examples.WordCountIT.testE2EWordCount(WordCountIT.java:69) 3 tests completed, 1 failed Finished generating test XML results (0.008 secs) into: Generating HTML test report... Finished generating test html results (0.009 secs) into: :beam-runners-google-cloud-dataflow-java-examples:preCommitFnApiWorker (Thread[Daemon worker,5,main]) completed. Took 1 hrs 2 mins 51.976 secs. :beam-runners-google-cloud-dataflow-java:cleanUpDockerImages (Thread[Daemon worker,5,main]) started. > Task :beam-runners-google-cloud-dataflow-java:cleanUpDockerImages Caching disabled for task ':beam-runners-google-cloud-dataflow-java:cleanUpDockerImages': Caching has not been enabled for the task Task ':beam-runners-google-cloud-dataflow-java:cleanUpDockerImages' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'docker''. Working directory: Command: docker rmi us.gcr.io/apache-beam-testing/java-postcommit-it/java:20190103060025 Successfully started process 'command 'docker'' Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20190103060025 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:73905979282ce6deb6b178c43e4d38f13781d92ee95119527c7069f3ab5be33e Starting process 'command 'gcloud''. Working directory: Command: gcloud --quiet container images delete --force-delete-tags us.gcr.io/apache-beam-testing/java-postcommit-it/java:20190103060025 Successfully started process 'command 'gcloud'' Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:73905979282ce6deb6b178c43e4d38f13781d92ee95119527c7069f3ab5be33e Associated tags: - 20190103060025 Tags: - us.gcr.io/apache-beam-testing/java-postcommit-it/java:20190103060025 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20190103060025]. Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:73905979282ce6deb6b178c43e4d38f13781d92ee95119527c7069f3ab5be33e]. :beam-runners-google-cloud-dataflow-java:cleanUpDockerImages (Thread[Daemon worker,5,main]) completed. Took 2.199 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-google-cloud-dataflow-java-examples:preCommitFnApiWorker'. > There were failing tests. See the report at: file:// * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 5m 56s 92 actionable tasks: 86 executed, 5 from cache, 1 up-to-date Publishing build scan... https://gradle.com/s/bgu2feebn7wce Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org For additional commands, e-mail: builds-help@beam.apache.org