beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From echauc...@apache.org
Subject [beam] branch spark-runner_structured-streaming updated (3533779 -> 6392179)
Date Fri, 04 Jan 2019 10:38:22 GMT
This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.


 discard 3533779  Fix spotlessJava issues
 discard 77cacde  Add ReadSourceTranslatorStreaming
 discard 91baa65  Cleaning
 discard 36a72f7  Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
 discard d9869c4  Split batch and streaming sources and translators
 discard 8591d63  Run pipeline in batch mode or in streaming mode
 discard 87bec8e  Move DatasetSourceMock to proper batch mode
 discard 017dcb9  clean deps
 discard e510711  Use raw WindowedValue so that spark Encoders could work (temporary)
 discard 98d9049  fix mock, wire mock in translators and create a main test.
 discard 83f8487  Add source mocks
 discard 44fd6c7  Experiment over using spark Catalog to pass in Beam Source through spark Table
 discard e7ed784  Improve type enforcement in ReadSourceTranslator
 discard 8bcfa5f  Improve exception flow
 discard e86247f  start source instanciation
 discard 8e08c58  Apply spotless
 discard f54899b  update TODO
 discard 3e87c5e  Implement read transform
 discard b2d37bf  Use Iterators.transform() to return Iterable
 discard 7b00f7c  Add primitive GroupByKeyTranslatorBatch implementation
 discard 5ca19f2  Add Flatten transformation translator
 discard 59acff8  Create Datasets manipulation methods
 discard 26238ce  Create PCollections manipulation methods
 discard 9061cb0  Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
 discard 463178b  Added SparkRunnerRegistrar
 discard 358170a  Add precise TODO for multiple TransformTranslator per transform URN
 discard c567d40  Post-pone batch qualifier in all classes names for readability
 discard af6a350  Add TODOs
 discard 74054fe  Make codestyle and firebug happy
 discard 92ed130  apply spotless for e-formatting
 discard 41742be  Move common translation context components to superclass
 discard fc03a4a  Move SparkTransformOverrides to correct package
 discard 3251ab2  Improve javadocs
 discard 8184de0  Make transform translation clearer: renaming, comments
 discard 3ce792d  Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
 discard 53b2e71  Initialise BatchTranslationContext
 discard 5cb1add  Organise methods in PipelineTranslator
 discard b6b426e  Renames: better differenciate pipeline translator for transform translator
 discard b77c7bb  Wire node translators with pipeline translator
 discard 91f283a  Add nodes translators structure
 discard 999761a  Add global pipeline translation structure
 discard ad32304  Start pipeline translation
 discard 8b5c33c  Add SparkPipelineOptions
 discard 0a89b75  Fix missing dep
 discard 03d333d  Add an empty spark-structured-streaming runner project targeting spark 2.4.0
     add cb4358c  [BEAM-6098] Support lookup join symmetric in left/right inputs
     add b484402  Merge pull request #7118: [BEAM-6098] Support lookup join symmetric in left/right inputs
     add 7822091  [BEAM-6082] Fix enum for SQL query 5 and 7
     add 9d05eed  Merge pull request #7133: [BEAM-6082] Fix enum for SQL query 5 and 7
     add 7963df4  Updates the script for cutting a release branch
     add bedd4d9  Merge pull request #7108: Updates the script for cutting a release branch
     add e662b5f  [BEAM-6102] Legacy-worker gradle changes
     add adf659e  [BEAM-6114] Add isBounded() to BeamRelNode and BeamSqlTable, use for JOIN
     add 048471b  Merge pull request #7121: [BEAM-6114] Add isBounded() to BeamRelNode and BeamSqlTable, use for JOIN
     add 148808a  [BEAM-6102] Move the worker jar before the pipeline jar when using the --dataflowWorkerJar option (#7143)
     add 02c763b  [BEAM-5197] Fix UnboundedSourceWrapper#testWatermarkEmission (#7138)
     add f84feac  [BEAM-6058] Adding option for flink configuration directory and setting config in exectution environment
     add c2aaf2d  Merge pull request #7031: [BEAM-6058] Adding option for Flink configuration directory and setting config in execution environment
     add d922f70  [BEAM-6102] Fix several packages that were being bundled without relocation within the Dataflow worker. (#7145)
     add 4082ce5  [BEAM-2687] Correctly handle read-before-write semantics for user state. (#7102)
     add 01ef416  Add argument parsing and filtering to coders microbenchmark.
     add b76f38b  Missing type declaration in default coder.
     add 45c114d  Optimize sequence coders.
     add 18fde2e  Restore unicode typing after Py3 changes.
     add 0a33d9b  Optimize LengthPrefixCoder.
     add eb4a65a  Optimize WindowedValue encoding.
     add 1cc6efc  Optimize IntervalWindow encoding.
     add 06d4a56  Merge pull request #7130 from Optimize Python coders.
     add 400c174  BEAM-6134: MongoDbIO add support for projection
     add 62b63fc  Merge pull request #7148 from chaimt/BEAM-6134
     add 68be16a  [BEAM-6100] Collect metrics properly in Load tests (#7087)
     add 3fc46b5  [BEAM-5197] Fix possible overflow of timestamp for processing timer
     add 0261783  Add an option to create Dataflow piplines from a snapshot
     add 55fae0d  Fix merge conflicts
     add b9e6fa1  Merge pull request #7092 from dpmills/snapshots
     add b1c0993  Update Java Container beam-master-20181128
     add 05a7801  Undo accidental commit b1c09939a54527ca47d02daf742001012a2be149
     add 8aff81d  Update Dataflow worker container to beam-master-20181128
     add df23d47  Merge pull request #7154 from charlesccychen/update-container
     add de3bd4af Future-proofing for TensorFlow2 (#7155)
     add 004cc54  Revert "Merge pull request #7051 from markflyhigh/py-move-off-apitools-1"
     add d027d96  Merge pull request #7156: [BEAM-6145] Revert "Merge pull request #7051 from markflyhigh/py-move-off-apitools-1"
     add 0b7dc3d  [BEAM-6148] ptest support arbitrary runners
     add 64a6a3c  Merge pull request #7158 from lostluck/ptest
     add a440344  [BEAM-4726] Add heap profiling hook
     add d9ae90f  Merge pull request #7159 from lostluck/heap
     add 0a45041  [BEAM-3612] Closurize method invocations
     add 1024472  Merge pull request #7161 from lostluck/wrap
     add 6cd0d9b  [BEAM-3661] Port TriggerExampleTest off DoFnTester
     add 2b5a604  Merge pull request #7125: [BEAM-3661] Port TriggerExampleTest off DoFnTester
     add b629727  [SQL] Add support for TableMacro UDF
     add 2f0f3f2  Merge pull request #7141 from kanterov/kanterov_table_functions
     add b8d2423  Fix Go lints on Initialized
     add 96fe92d  Merge pull request #7167 from lostluck/fixtypos
     add 4a6527d  Revert "Optimize several Python coder implementations."
     add fc373df  Merge pull request #7166 from apache/revert-7130-fast-coders
     add 5b9641d  [BEAM-5978] Use dynamic port when starting auto starting jobserver
     add 38e6be9  [BEAM-5978] Adding libltdl7 to flink job server docker
     add 65136cf  [BEAM-5978] Correctly pick the docker executable
     add 63a4c18  [BEAM-5978] Increase portable wordcount threads to avoid dead lock
     add 49c8386  [BEAM-6146] Add precommit for portable python
     add 5506335  [BEAM-6146] Portable Python Precommit test
     add b06b8e5  Merge pull request #6954: [BEAM-6146] Add portable WordCount to Python PreCommit
     add 4ca0e89  [BEAM-3659] Port UserScoreTest off DoFnTester
     add bd81e8b  Merge pull request #7126: [BEAM-3659] Port UserScoreTest off DoFnTester
     add d136637  [BEAM-2939] SplittableDoFn Java SDK API Changes (#6969)
     add e55c514  [BEAM-6111] Fix flaky PortableTimersExecutionTest (#7171)
     add ac10c91  Add packageDeclaration Checkstyle check + fix an issue it threw up (#7172)
     add ad0cb4f  Simplifying a few junit assertions (#7139)
     add d116dd3  [BEAM-6162] Fix PipelineOptions argparse behavior
     add 1768511  Merge pull request #7176 from charlesccychen/fix-argparse
     add a9dff65  Merge pull request #7175: [BEAM-5884] Move the nullable attribute onto FieldType.
     add 6119242  [BEAM-6163] Build python boot for mac and support process env on mac
     add 6517025  Merge pull request #7178: [BEAM-6163] Build python boot for mac and support process env on mac
     add 60cc4c1  Merge pull request #7147 : [BEAM-4453] Use constructors to generate schema POJOs
     add 42f39e9  [BEAM-6143] Upgrade to Mongo Java Driver 3.9.1 and update the API
     add b469880  Merge pull request #7151: [BEAM-6143] Upgrade to Mongo Java Driver 3.9.1 and update the API
     add f2d1581  [BEAM-5396] Savepoint restore option in Flink runner.
     add 236d0dd  Merge pull request #7169: [BEAM-5396] Savepoint restore option in Flink runner
     add 3432c04  [BEAM-6032] Move PortableValidatesRunner configuration out of BeamModulePlugin (#7173)
     add ff93af2  [BEAM-5925] Add .withSocketAndRetryTimeout() and .withConnectTimeout() to ElasticseachIO.ConnectionConfiguration
     add a566874  [BEAM-5925] Set a socket and retry timeout of 1.5min and a connect timeout of 5s for all connections in all elasticsearch unit tests"
     add d98c9f4  Merge pull request #7065 from wscheep/elastic_maxretrytimeout
     add aeeb085  [BEAM-5984] Enable publishing load test results to BigQuery
     add 9d3daf2  [BEAM-5984] Use more high level method for creating BQ rows in Nexmark too
     add ab3519e  [BEAM-5984] Provide generic BigQueryResultPublisher class
     add 90ef4eb  Merge pull request #7090: [BEAM-5984] Enable publishing load test results to Big Query
     add 4c97d0c  [BEAM-6146] Run pre-commit wordcount in batch and streaming mode.
     add b917a86  Merge pull request #7180: [BEAM-6146] Run pre-commit wordcount in batch and streaming mode
     add f040a4e  [BEAM-5859] Improve operator names for portable pipelines
     add e2e31c9  Merge pull request #7150: [BEAM-5859] Improve operator names for portable pipelines
     add aa4d9bc  Add more AVRO utilities to convert between Beam and Avro. Add schema-conversion utilities as well as a conversion from a Beam Row into GenericRecord.
     add 6b7cf42  Merge pull request #7181 : [BEAM-4454] Add more AVRO utilities to convert between Beam and Avro.
     add 78c1a10  [BEAM-6160] Use service server rather than service (#7168)
     add 270ef6a  [BEAM-2939] Add support for backlog reporting to byte key and offset restriction trackers. (#7177)
     add 3494a8f  [BEAM-5058] Run basic ITs in Python Precommit in parallel
     add 3a348e8  Merge pull request #7163 from markflyhigh/py-precommit-it
     add 34420cc  [BEAM-6174] Kryo dependency removed.
     add 54214f8  Merge pull request #7194: [BEAM-6174] Kryo dependency removed.
     add 2aa5d07  [BEAM-5778] Add integrations of Metrics API to Big Query for SyntheticSources load tests in Python SDK
     add ee515bf  Merge pull request #6943: [BEAM-5778] Add integrations of Metrics API to Big Query for SyntheticcSources load tests in Python SDK
     add 08dafbe  [BEAM-1628] Allow empty port for flink master url
     add bec7dac  Merge pull request #7187: [BEAM-1628] Allow empty port for flink master url
     add 9c018ac  [BEAM-6122] Update committer guidelines
     add 1c67861  Address review comments.
     add 702b9de  Move squash paragraph under merging.
     add e5d9cf4  Merge pull request #7129: [BEAM-6122] Update committer guidelines
     add 9593adb  [BEAM-6077] If available, use max_parallelism for splitting unbounded source
     add 5565b0a  [BEAM-6077] Tests for read source translator
     add a3a8a32  Merge pull request #7128: [BEAM-6077] If available, use max_parallelism for splitting unbounded source
     add 74ed7ac  [BEAM-5462] get rid of <pipeline>.options deprecation warnings in tests
     add 95d0ac5  Merge pull request #6930: [BEAM-5462] get rid of <pipeline>.options deprecation warnings in tests
     add a3d2611  [BEAM-2400] Use getJobId() consistently
     add a5b36c5  Merge pull request #7199: [BEAM-2400] Use getJobId() consistently
     add 2f9330c  [BEAM-6180] Remove duplicated IdGenerator from runner harness and use IdGenerator from fnexecution instead. (#7201)
     add 681b5cd  Merge pull request #7204:  [BEAM-5884] Fix FieldType comparison in BeamSQL
     add 33453c2  Revert "Revert "Optimize several Python coder implementations.""
     add 50d8392  [BEAM-6153] Stricter interval window comparison.
     add e7ab8c4  Merge pull request #7170 from [BEAM-6153] Re-enable coder optimization.
     add 43fe997  [BEAM-5817] Add SQL bounded side input join to queries that are actually run
     add a3510e0  Merge pull request #7205: [BEAM-5817] Add SQL bounded side input join to queries that are actually run
     add d897c5c  BEAM-6151: MongoDbIO add support mongodb server with self signed ssl
     add 9fbe80e  Merge branch 'master' into BEAM-6151
     add 385f2a1  Merge pull request #7162 from chaimt/BEAM-6151
     add 9448dba  [BEAM-6182] Disable conscrypt by default (#7203)
     add 8f15b88  [BEAM-3912] Add HadoopOutputFormatIO support
     add 406f8d7  [BEAM-3912] Remove useless dep
     add 86f723e  [BEAM-3912] Add HadoopOutputFormatIO support
     add 9863c79  [BEAM-3912] Remove useless dep
     add fa9cc4a  Fix typo in test name
     add 757b71e  [BEAM-3912] Implement HadoopFormatIO.Write
     add 20e3e24  [BEAM-5309] Add streaming support for HadoopFormatIO
     add 4adc254  [BEAM-5309] Add streaming support for HadoopFormatIO
     add aec6d82  Merge pull request #6691: [BEAM-5309] Add streaming support for HadoopFormatIO
     add af05ee2  Add portable-runner dependency to wordcount example as one of the defaults.
     add 736077c  Merge pull request #7213 from [BEAM-6184] Add portable-runner dependency to example pom.xml
     add 60da04a  [BEAM-5859] Better handle fused composite stage names.
     add 5ae80df  Merge pull request #7208: [BEAM-5859] Better handle fused composite stage names.
     add 0edc85e  [BEAM-6067] In Python SDK, specify pipeline_proto_coder_id property in non-Beam-standard CloudObject coders (#7081)
     add ecc2d84  Fixup User_COUNTER_URN_PREFIX to contain the trailing: (#7188)
     add 75d45e2  [BEAM-6159] Make Dataflow worker use ExecutableStage to process bundle (#7015)
     add a466104  Update PortableTimersExecutionTest to use PAssert, to prevent a concurrency issue collecting the test results.
     add 79df784  Remove extra timeout code from PortableTimersExecutionTest.
     add ce15b25  Merge pull request #7214: Update PortableTimersExecutionTest to use PAssert
     add 5e94da3  [BEAM-5920] Add additional owners for Community Metrics
     add 5850c00  Merge pull request #7186: [BEAM-5920] Add additional owners for Community Metrics
     add bc859cc  [BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow Runner. (#7202)
     add 9159d9b  [BEAM-5321] Port transforms package to Python 3 (#7104)
     add 4cd1226  Add QueueingBeamFnDataClient and make process, finish and start run on the same thread to support metrics. (#6786)
     add 1f6dd22  [BEAM-6155] Updates the GCS library the Go SDK uses.
     add ca6ee63  Merge pull request #7182 from bramp/BEAM-6155
     add 4d63cd3  [BEAM-5167] Log unscheduled process bundle requrests
     add e6add6a  Merge pull request #7192 from [BEAM-5167] Log unscheduled process bundle requests
     add fceeaef  Removing some unnecessary parentheses
     add 64c62b1  Merge pull request #7185 Removing some unnecessary parentheses
     add ee0801f  Enabling the ArrayTypeStyle checkstyle module
     add ec5602f  Merge pull request #7062 Enabling the ArrayTypeStyle checkstyle module
     add 61e8106  Fix translate_pattern test on Python 3.7
     add 40977f4  Merge pull request #6739 [BEAM-5787] Fix test_translate_pattern on Python 3.7
     add c314dfe  Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to provide specs and validate MonitoringInfos are properly populated.
     add a21b196  [BEAM-6194] Follow up with cleanup for https://github.com/apache/beam/pull/7015 (#7219)
     add c3102a9  Fix precommits due to concurrent submissions.
     add aa09bbb  Merge pull request #7230 Fix precommits due to concurrent submissions.
     add ab59d6d  [BEAM-6139] Adding support for BQ GEOGRAPHY data type (#7144)
     add ac3a9df  Redirect from nexmark page's older location
     add a226343  Merge pull request #7225 from udim/patch-2
     add 17968a2  [BEAM-6195] Make ProcessRemoteBundleOperation map PCollectionId into correct OutputReceiver and throws Exception when there is more than one input PCollection. (#7223)
     add 378d907  [BEAM-6167] Add class ReadFromTextWithFilename (Python) (#7193)
     add 36e3f98  [BEAM-4150] Use unwindowed coder in FnApiRunner optimization phases.
     add 6e1a8cd  [BEAM-6186] Optimization cleanup: move phase utilities out of local scope.
     add a076444  Merge pull request #7227 from [BEAM-6186] Optimization cleanup
     add e147bf9  [BEAM-6120] Support retrieval of large gbk iterables over the state API.
     add c2a9cac  Merge pull request #7127 from [BEAM-6120] Large gbk iterables
     add c3e636d  Add instructions to post-commit policy web page, according to discussions in dev mailing list.
     add f976430  Update website/src/contribute/postcommits-policies-details.md
     add 6e3cf84  Merge pull request #7095 from HuangLED/update_postcommit_doc
     add 2b95624  Move string literals to the left hand side of the expression in a few places
     add e7b2f30  Merge pull request #6887 from coheigea/string_literals
     add 2905227  Clarify usage of PipelineOptions subclass
     add e963882  Merge pull request #6872 Clarify snippet for PipelineOptions subclass
     add d4fd5a1  [BEAM-5866] Override structuralValue in ListCoder
     add 8b4f60e  [BEAM-5866] Override structuralValue in MapCoder
     add 7a394cf  Merge pull request #6862 from [BEAM-5866] structuralValue in List/MapCoder
     add 8783994  [BEAM-4444] Parquet IO for Python SDK (#6763)
     add a07da9e  Upgrade to Apache Tika 19.1
     add 1c6f145  Merge pull request #6719 Upgrade to Apache Tika 1.19.1
     add 8100f32  [BEAM-6079] Add ability for CassandraIO to delete data
     add 07d9311  [BEAM-6079] Fix access level and clean up generics issues
     add b0aae58  Merge pull request #7064: [BEAM-6079] Add ability for CassandraIO to delete data
     add 010357d  [BEAM-3657] Port JoinExamplesTest off DoFnTester
     add 29a7917  Merge pull request #7179: [BEAM-3657] Port JoinExamplesTest off DoFnTester
     add f8ef83b  [BEAM-5514] BigQueryIO doesn't handle quotaExceeded errors properly
     add b592f94  Merge pull request #7189: [BEAM-5514] BigQueryIO doesn't handle quotaExceeded errors properly
     add dc0946f  [BEAM-6201] Move fromJsonString() method to SyntheticOptions class
     add cd8f014  [BEAM-6201] Add SyntheticDataPubSubPublisher pipeline
     add b0069eb  Merge pull request #7238: [BEAM-6201] Data insertion pipeline
     add eb9065b  [BEAM-6176] Support IPv6 addresses for Flink master url
     add 4d65d72  Merge pull request #7196: [BEAM-6176] Support IPv6 addresses for Flink master url
     add f57def8  [BEAM-6172] Adjust Flink metric names / Add metric reporting tests
     add d80957c  Merge pull request #7207: [BEAM-6172] Adjust Flink metric names / Add metric reporting tests
     add 57c248b  Remove trailing whitespace.
     add 0d2ec39  Quiet c-extension-no-member lint warning.
     add edc37f9  Merge pull request #7249: Fix various lint errors in Python presubmit.
     add a9709ff  [BEAM-6186] Move combiner lifting out of line.
     add 3398a88  [BEAM-4678] Support combiner lifting in portable Flink runner.
     add d4237ec  Merge pull request #7228 from [BEAM-4678] Combiner lifting in portable Flink.
     add bfd1be9  [BEAM-6178] Adding beam-sdks-java-bom, adding exportJavadoc flag for applyJavaNature (#7197)
     add 17a881b  [BEAM-6181] Unexpected metrics should be non-fatal.
     add 4160b67  Merge pull request #7250 from [BEAM-6181] Unexpected metrics non-fatal.
     add cb06639  [BEAM-6033] normalize httplib2.Http initialization and usage
     add 3a182d6  Merge pull request #7032: [BEAM-6033] normalize httplib2.Http initialization and usage
     add fddb684  [BEAM-5058] remove unused envdir parameter
     add e8cbbf6  Merge pull request #7246: [BEAM-5058] remove unused envdir parameter
     add ebe81b9  [BEAM-2943] Fix typo in filename on FlinkRunner page
     add 8c8b7c3  Merge pull request #7254: [BEAM-2943] Fix typo in filename on FlinkRunner page
     add 73a4325  Use environment to control worker startup in FnApiRunner.
     add efd261a  [BEAM-6094] Implement external environment for Portable Beam.
     add ec55000  Merge pull request #7078 from [BEAM-6094] External portable BeamPython.
     add 9853bd7  [BEAM-6213] Fix matching of glob patterns in windows local filesystem
     add 289d2b2  Merge pull request #7258: [BEAM-6213] Fix matching of glob patterns in windows local filesystem
     add c00dfa5  Fix combiner lifting for external job service.
     add 16b34cf  Merge pull request #7260 Fix combiner lifting for external job service.
     add bc11c06  [BEAM-6216] Update flinkMaster URL in the nexmark wep-page to reflect change in FlinkExecutionEnvironments.
     add ffcd9f4  [BEAM-6216] Update flinkMaster URL in nexmark postCommit script to reflect change in FlinkExecutionEnvironments.
     add a53f56a  Merge pull request #7261 from echauchot/BEAM-6216-flink-local
     add 9e8ac83  [BEAM-6240] Clean up duplicated SocketAddressFactory class.
     add 798b3b3  Merge pull request #7241: [BEAM-6204] Clean up duplicated SocketAddressFactory class.
     add 5e506bf  [BEAM-6205] Setup gradle task ro run fnapi worker test with use_executable_stage_bundle_execution
     add 5574f47  Merge pull request #7243: [BEAM-6205] Setup gradle task ro run fnapi worker test with use_execu…
     add 8828b16  Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums
     add a61f2c5  Merge pull request #7242: Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums
     add 11513c3  Merge remote-tracking branch 'upstream/master' into pr7244
     add dc10f75  Merge pull request #7244: [BEAM-6138] Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro…
     add 41719ac  Fixing publishing problem introduced in #7197
     add 20df151  Merge pull request #7265: Fixing publishing problem introduced in #7197
     add 07bae67  [BEAM-5419] Add Flink multi-version build layout
     add 6308a6d  [BEAM-5267] Make source Flink v1.6 compatible
     add d0254f0  [BEAM-5267] Add Flink v1.6 target for Flink Runner
     add 3400ba6  Merge pull request #7229: [BEAM-5419] Add Flink multi-version builds
     add 2a40c57  Add remaining Schema support for AVRO records:   * Add support for SpecificRecord using ByteBuddy codegen.   * Add helper methods for GenericRecord.   * Fix uncovered bugs involving nullable support.
     add 0a74b17  Add period to sentence.
     add 8b20602  Merge pull request #7233:  [BEAM-4454] Add remaining functionality for AVRO schemas
     add 3fafec1  [BEAM-5320] [BEAM-6106] Finish Python 3 porting for testing module (#7262)
     add bd0103c  [BEAM-5978] Changing parallalim for wordcount to 1
     add 1518361  Merge pull request #7174: [BEAM-5978] Changing parallelism for wordcount to 1
     add a89d296  Fix broken beam-sdks-python:test
     add 4eb7744  Merge pull request #7273: Fix broken beam-sdks-python:test
     add 0b3b9e0  [BEAM-6138] Add User Counter Metric Support to Java SDK (#6799)
     add 1775c19  Stop subclassing user pojos.
     add 6280255  spotless.
     add f8f9ca5  Merge pull request #7234:  [BEAM-4453] Stop subclassing user POJOs.
     add e2db4d5  Updates Beam Website to use 2.9.0 as the latest release
     add 0f14b40  Merge pull request #7215: Updates Beam Website to use 2.9.0 as the latest release
     add daed1e6  Blog for Apache Beam 2.9.0 release
     add 5df8cb2  Merge pull request #7275: Blog for Apache Beam 2.9.0 release
     add 2cac7ba  Updates blog for 2.9.0 release
     add e21b80a  Merge pull request #7278: Updates blog for 2.9.0 release
     add eae5521  Add reshuffle option to Create.
     add a34e459  Merge pull request #7274 Add reshuffle option to Create.
     add 0d60a89  [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime
     add 88f181c  Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime
     add fc38359  [BEAM-6227] Fix GroupByKey with null values in Flink Runner
     add 2886473  Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner
     add 788ce61  Merge pull request #7267: [BEAM-4454] Support Avro POJO objects
     add 54e2fc1  [BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for specific failing http calls. Plus, add a custom error message in BigQueryServicesImpl. (#7270)
     add 69358c5  [BEAM-6191] Remove redundant error logging for Dataflow exception handling
     add 977080f  Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling
     add f6c1dd5  [BEAM-6179] Fixing bundle estimation when all xs are same
     add 71890da  Merge pull request #7280 from angoenka/fix_bundle_estimation
     add 5ec695b  [BEAM-6150] Superinterface for SerializableFunction allowing declared exceptions (#7160)
     add 4f90294  Remove Gradle from Jenkins job names
     add ec3f792  Merge pull request #7286 from swegner/jenkins_gradle
     add b7035c1  [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage (#7271)
     add 53a5ce7  [BEAM-4594] Support state continuation tokens over the Fn API.
     add 39721d2  Merge pull request #7252 from [BEAM-4594]  State continuation tokens.
     add f9bc485  Mention portable Flink runner support for state and timers in 2.9.0 release blog
     add bd5bbf9  Merge pull request #7294: Mention portable Flink runner support for state and timers in 2.9.0 release blog
     add c26d532  More robust FnApi Runner.
     add ca3eb14  Merge pull request #7251 from More robust FnApi Runner.
     add e4f6517  [BEAM-6227] Do not compare recovered state against structural null value
     add e108cca  Merge pull request #7291: [BEAM-6227] Do not compare recovered state against structural null value
     add 8a333e7  [BEAM-6235] Upgrade AutoValue to version 1.6.3
     add 4588c25  Merge pull request #7285: [BEAM-6235] Upgrade AutoValue to version 1.6.3
     add 89ad88d  [BEAM-2873] Setting number of shards for writes with runner determined sharding
     add a68f209  Merge pull request #4760: [BEAM-2873] Setting number of shards for writes with runner determined sharding
     add a6d4345  [BEAM-6200] Deprecate old HadoopInputFormatIO, move it into new HadoopFormatIO
     add 67d0f78  Merge pull request #7263: [BEAM-6200] Deprecate old HadoopInputFormatIO, move it into new HadoopFormatIO
     add 2a5cc73  [BEAM-5419] Simplify job-server-container targets
     add 1061dc9  Merge pull request #7299: [BEAM-5419] Simplify job-server-container targets
     add da7fec2  [BEAM-6190] Add processing stuck message to Pantheon.
     add b070e89  Fixed some style errors and tests that needed an additional parameter.
     add fa522c5  Merge pull request #7240 from dustin12/lullError
     add 2555945  Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
     add dc62028  [BEAM-6067] Update BeamPython side-input support in the Dataflow runner for the unified worker. (#7269)
     add bc1609d  [BEAM-6170] Change Nexmark stuckness warnings to not cause pipeline failure.
     add c29092e  Merge pull request #7191 from scwhittle/remove_stuck_error
     add 0cdfca4  Avoid creating a variable and immediately returning it.
     add af5d7bf  Merge pull request #7007 from coheigea/return
     add eab1d0e  [BEAM-6252] SamzaRunner: Add a registrar to allow customized DoFnInvoker
     add 6a6fbe4  Allow Samza DoFnInvoker to generate configs and pass in TaskContext
     add 319a370  Merge pull request #7301: [BEAM-6252] SamzaRunner: Add a registrar to allow customized DoFnInvoker
     add 473e5a6  [BEAM-6253] SamzaRunner: Add a few customized transforms for runner use cases
     add 6ba3a45  Merge pull request #7302: [BEAM-6253] SamzaRunner: Add a few customized transforms for runner use cases
     add 2563e92  Update Dataflow Python container to 20181218
     add 335fdb2  Merge pull request #7306 from charlesccychen/update-container
     add a7315c5  [BEAM-6197] Log time for Dataflow GCS upload of staged files
     add 095ce13  Merge pull request #7235 from alanmyrvold/uploadGCS
     add df4b623  [BEAM-5167] Ensure monitoring thread does not prevent process exit.
     add d3fbf80  Merge pull request #7248 from robertwb/monitor-thread
     add 4573c56  Fix time format on Windows.
     add e4577ed  Merge pull request #7257 Fix time format on Windows.
     add 0ef5e66  Update release guide with new Jenkins job name.
     add aa0fdc5  Merge pull request #7311: Update release guide with new Jenkins job name.
     add 384de82  WEBSITE: update community nav, add in-person page
     add 5845e2a  Merge pull request #7314: WEBSITE: update community nav, add in-person page
     add b5223f4  [BEAM-6263] Prevent NullPointer on concurrent JobServer shutdown
     add a50ae05  [BEAM-6263] Prevent port collisions in FlinkJobServerDriveTest
     add 5d3b420  [BEAM-6263] Prevent stderr->stdout redirection
     add 19f7dfa  [BEAM-6263] Restore stderr on exceptions to print the error
     add 302b883  Merge pull request #7309: [BEAM-6263] Fix error-prone test setup for FlinkJobServerDriver
     add 6ff3129  [BEAM-6186] Finish moving optimization phases.
     add dba30b4  Merge pull request #7281 from [BEAM-6186] Finish moving optimization phases.
     add 6de7cb7  [BEAM-6094] Add loopback environment for java.
     add a49c835  More complete exception message.
     add 66ff825  Merge pull request #7307 [BEAM-6094] Add loopback environment for java.
     add 6e2ca58  [BEAM-6245] Set translation mode directly on PipelineOptions
     add f67ac59  [BEAM-6245] Add integration test for FlinkTransformOverrides
     add 85bd95e  Merge pull request #7296: [BEAM-6245] Add integration test for FlinkTransformOverrides
     add 8214cb6  [BEAM-5449] Tagging failing ULR ValidatesRunner tests.
     add bd68ef6  Merge pull request #7295: [BEAM-5449] Tagging failing ULR ValidatesRunner tests.
     add 0c589db  [BEAM-5993] Create SideInput Load test (#7020)
     add 61d7e53  Fix documentation for PCollection Type in XMLIO example
     add f036699  Merge pull request #7320: [BEAM-6270] Fix documentation for PCollection Type in XmlIO example
     add c0655d1  Rename v1_13_1 to v1p13p1.
     add 85ea7ba  Depend on local vendored jar instead of the one from Maven
     add 2afc42b  Disable 'jar' task for vendoring projects.
     add a5a139d  Merge pull request #7324: [BEAM-6056] Rename vendored guava relocation to v1p13p1
     add 33c85e1  [BEAM-6268] Ignore failed HadoopFormatIOCassandraTest (#7325)
     add 8850ad7  Python 3 port io.filesystem module
     add 04a647d  Merge pull request #7318 from RobbeSneyders/filesystem
     add 7217b47  [BEAM-6273] Update dependencies pages with 2.9.0
     add 230c7c9  Merge pull request #7327: [BEAM-6273] Update dependencies pages with 2.9.0
     add 9a51eb3  [BEAM-6179] Fixing itterable comparison
     add d9e318c  Fix whitespace.
     add bed5747  Merge pull request #7313 from angoenka/fix_bundle_estimation
     add 7248602  [BEAM-6262] KinesisIO - gracefully shutdown executor service
     add e9d51ec  Merge pull request #7315: [BEAM-6262] KinesisIO - gracefully shutdown executor service
     add 5d305af  Fix performance regression.
     add c039440  Merge pull request #7331: [BEAM-6276] Fix performance regression.
     add 6c39b61  Adds a link to release notes for Beam 2.9.0.
     add 7c8f7ba  Merge pull request #7329: Adds a link to release notes for Beam 2.9.0
     add 1e41220  [BEAM-6165] Send metrics to Flink in portable Flink runner (#7183)
     add e07374a  [BEAM-5334] Remove unused 'language' argument
     add 4290921  Merge pull request #7264 from udim/perf-tests
     add 9d4302d  [BEAM-5539] Beam Dependency Update Request: google-cloud-pubsub
     add 0644c55  Merge pull request #7268 from ihji/upgrade_pubsub
     add b44ecb4  [BEAM-6286] Add SamzaRunner profile to mvn archetype
     add fb4b151  Merge pull request #7335: [BEAM-6286] Add SamzaRunner profile to mvn archetype
     add 8d611b4  [BEAM-6212] Add MongoDbIO ordered option
     add 094586d  Merge pull request #7256: [BEAM-6212] Add MongoDbIO ordered option
     add 7302aef  Update Slack invitation on #general vs #beam channel
     add 845de99  [BEAM-6283] Convert PortableStateExecutionTest and PortableExecutionTest to using PAssert
     add f1e339b  [BEAM-6295] Fix versions in 2.8.0 Java dependencies table
     add 941cb27  Merge pull request #7341 from melap/dependencies
     add e6e85ed  Reimplement GCS copies with rewrites.
     add b196397  Merge pull request #7050: [BEAM-5959] Reimplement GCS copies with rewrites.
     add 24aa20f  Add toplevel :sqlPostCommit gradle command
     add 0f3560a  Add Jenkins job to run :sqlPostCommit
     add 0b88bca  Merge pull request #7338: [BEAM-6288] Add SQL postcommit
     add 80b0c6a  [BEAM-6295] Fix versions in 2.7.0 Java dependencies table
     add 76b180b  Merge pull request #7342 from melap/dependencies
     add 1c2d631  Update JUnit
     add ad43619  Merge pull request #7344: [BEAM-6299] Update JUnit to fix bug with parameterized tests
     add 2f5ba05  Put generated getter/setter/creator classes in the same package as the class they are modified.
     add a02d884  Merge pull request #7345: [BEAM-6300] Put generated getter/setter/creator classes in the same package as the class they access
     add 8bdbb33  Add schema support to AvroIO and PubsubIO. For backwards-compatibility reasons, Beam schema support must be explicitly enabled in these sources.
     add 4f23004  Remove unneeded @Rule.
     add 2681c25  Merge pull request #7290: [BEAM-4454] Support avro schema inference in sources
     add a90dabf  [website] Point Slack link to #beam channel instead of #general
     add 9460fee  Merge pull request #7346: [website] Point Slack link to #beam channel instead of #general
     add 77791da  [BEAM-6239] Add session side input join to Nexmark
     add ac8c956  Merge pull request #7287: [BEAM-6239] Add session side input join to Nexmark
     add 0ad4a5d  [BEAM-6244] Restore updateProducerProperties
     add 9b0d8fb  [BEAM-6244] Restore validate
     add e636294  Merge pull request #7343: [BEAM-6244] KafkaIO: keep KafkaIO.Write compatibility with 2.9.0
     add 45a61e4  BEAM-6306 Upgrade Jackson to version 2.9.8
     add ba01b8e  Merge pull request #7352: [BEAM-6306] Upgrade Jackson to version 2.9.8
     add 4ca3cf0  Upgrade to Calcite 1.18
     add 718aef7  Merge pull request #7209 from apilloud/upgrade
     add 8588d52  Add time usage in seconds for staging files.
     add 41eeb39  Merge pull request #7336: Add time usage in seconds for staging files.
     add 6439fb1  Python 3 port io.filesystemio module
     add d5638e7  Add apache_beam.io.localfilesystem_test to python 3 test suite
     add 7710391  Merge pull request #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module
     add be7549c  [BEAM-6287] pyarrow is not supported on Windows Python 2
     add dc01009  Merge pull request #7337: [BEAM-6287] pyarrow is not supported on Windows Python 2
     add bac909b  Treat VarInt encoding as a Beam primitive encoding in Dataflow runner (#7351)
     add f720985  [BEAM-6110] For SQL CoGBK-based join use EARLIEST output timestamp
     add f190152  Merge pull request #7115 from kennknowles/sql-join-cogbk-timestamps
     add ffec485  Flink 1.5.6 upgrade (#7322)
     add 671ed3f  Update data source for syncing jobs from Jenkins.
     add 8f50ac2  Update deployment versions.
     add 498b186  Merge pull request #7364 from Ardagan/FixBMetrics
     add e413099  [BEAM-4726] Add arity specialization for calling and returns.
     add a47b697  Merge pull request #7355 from lostluck/arity
     add 643e562  Enforce the checkstyle IllegalThrows rule for throwing Error + RuntimeException
     add 680d911  Merge pull request #7259 from coheigea/illegal_throws
     add 100c561  [BEAM-5918] Fix CastTest
     add d7c64e7  Merge pull request #7372: [BEAM-5918] Fix CastTest
     add 25865d3  [BEAM-5467] Increase test timeout for portable ValidatesRunner tests
     add d1384b9  Merge pull request #7376: [BEAM-5467] Increase test timeout for portable ValidatesRunner tests
     add 4e8a07b  [BEAM-6294] Use Flink rebalance for shuffle.
     add 9f2eb34  Merge pull request #7360 [BEAM-6294] Use Flink rebalance for shuffle.
     add 2481ee6  Disable BigQueryIO validation since datasets and tables are created during runtime.
     add 3a0f70e  Merge pull request #7368 from boyuanzz/fix_bq
     add eab6759  [BEAM-4725] Use unsafe to avoid small allocations to the heap.
     add 8f38b46  Merge pull request #7357 from lostluck/smallbuf
     add 54d3857  [BEAM-6325] Cast cross compile output from []byte to string for printing
     add 7c9babd  Merge pull request #7375 from lostluck/cast
     add fb7ae4f  [BEAM-5112] Generate code for BeamCalcRel DoFn
     add 1ad4aff  Remove Beam Interpreter
     add e398bee  Merge pull request #6417 from apilloud/codegen
     add d3a38f5  [BEAM-6316] Fix container image name for PreCommit PortableWordCount
     add 9ad1074  Merge pull request #7377: [BEAM-6316] Fix container image name for PreCommit PortableWordCount
     add 0d50a17  Fix go runtime break
     add ca4defe  Merge pull request #7379 from lostluck/fixbreak
     add ec6384a  [BEAM-6329] Address synchronization issue for portable timers (#7359)
     add 1a4db4b  [BEAM-5386] Prevent CheckpointMarks from not getting acknowledged
     add ea275e4  [BEAM-5386] Assert that source thread enters sleep instead of terminating
     add f56c86f  Merge pull request #7349: [BEAM-5386] Prevent CheckpointMarks from not getting acknowledged
     add 926361b  [BEAM-5386] Move assertion out of finally block to not swallow original exception
     add 3b8abca  Upgrade vendored gRPC artifact version to 0.2
     add 15aa88d  Merge pull request #7328: [BEAM-6056] Upgrade vendored gRPC artifact version to 0.2
     add 14781c7  [BEAM-6056] Source vendored grpc dependency from Maven central
     add a25b64d  Merge pull request #7388: [BEAM-6056] Source vendored grpc dependency from Maven central
     add 095870f  Python 3 port io.range_trackers
     add 359ddb9  Add io.restriction_trackers_test to Python 3 test suite
     add bca5c60  Merge pull request #7358 from RobbeSneyders/trackers
     add 5ce0933  Updates release validating to run LeaderBoard example using Dataflow Streaming Engine
     add 5dd597e  Merge pull request #7365: [BEAM-6249] Adds an Streaming Engine based test to release validation
     add 5cdf3a7  [BEAM-5315] Python 3 port io.source* and io.concat_source* modules (#7383)
     add 4b039e4  [BEAM-5315] Python 3 port io.filebased_* modules (#7386)
     add fc482f1  [BEAM-5959] Add performance testing for writing many files
     add 41dd6e1  Merge pull request #7266 from udim/cmek-perf
     add a24b1af  Move org.apache.beam.runners.samza.util.Base64Serializer to org.apache.beam.runners.core.serialization.Base64Serializer to be used by other runners
     add 3b8ae00  Fix visibility of deserialize method
     add 0783779  Add missing package-info
     add 4660895  Merge pull request #7384 from echauchot/Base64Serializer
     add c4590a0  split SerializablePipelineOptions into serialization utils and instance class.
     add 5130bcb  Merge pull request #7385 from echauchot/exposeSerializationSerializablePipelineOptions
     add a404cee  Add paddedcell fix to spotlessJava rules.
     add c148c35  Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules.
     add c028ebc  Upgrade html-proofer and dependencies to latest
     add 07c279a  Remove broken links to datatorrent.com
     add b09e721  Fix pydoc link to GoogleCloudOptions
     add fd5e321  Remove broken link to atrato.io
     add a79ef89  Fix link to internal anchor
     add 5466ac0  Remove stale exclusions from HTML link checker.
     add a2986cc  Merge pull request #7393: [BEAM-5662] Clean up website html-proofer config
     add b02f79f  Disable UsesMetricsPusher tests for direct-runner
     add f74c979  Fix SplittableDoFnTest#testBoundedness
     add 459e730  [BEAM-6352] Ignore tests using Watch PTransform
     add 26c73ef  [BEAM-6353] Fix TFRecordIOTest
     add 92a6c23  [BEAM-6354] Add timeout and ignore hanging tests
     add 55ffd97  Add :beam-runners-direct-java:needsRunnerTests to javaPreCommit
     add c591727  Merge pull request #7374: Add :beam-runners-direct-java:needsRunnerTests to javaPreCommit
     add 3948595  [BEAM-5959] Reorder methods according to convention
     add 5716dba  Merge pull request #7403 from udim/cmek-perf
     add 5212b71  [BEAM-6030] Split metrics related options out of PipelineOptions
     add 185cb1a  [BEAM-6030] Add Experimental label on MetricsOptions
     add bd80118  Merge pull request #7400 from echauchot/BEAM-6030-metrics-sinks-pipelineOptions
     new ce39f93  Add an empty spark-structured-streaming runner project targeting spark 2.4.0
     new 737af2f  Fix missing dep
     new 018c773  Add SparkPipelineOptions
     new 1c97788  Start pipeline translation
     new abf4b46  Add global pipeline translation structure
     new 28a9422  Add nodes translators structure
     new 3a743c2  Wire node translators with pipeline translator
     new 051e8dc  Renames: better differenciate pipeline translator for transform translator
     new 6695d64  Organise methods in PipelineTranslator
     new ec9d634  Initialise BatchTranslationContext
     new 476cae8  Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
     new ce484e9  Make transform translation clearer: renaming, comments
     new 0033f89  Improve javadocs
     new 26f2e4b  Move SparkTransformOverrides to correct package
     new 4777e22  Move common translation context components to superclass
     new 0cfa70d  apply spotless for e-formatting
     new 91f9ef5  Make codestyle and firebug happy
     new 901a1ac  Add TODOs
     new 2ccccdd  Post-pone batch qualifier in all classes names for readability
     new b37da3e  Add precise TODO for multiple TransformTranslator per transform URN
     new bbf583c  Added SparkRunnerRegistrar
     new 866ef13  Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
     new 7a645e1  Create PCollections manipulation methods
     new 31fb182  Create Datasets manipulation methods
     new 286d7f3  Add Flatten transformation translator
     new 9e6fc2c  Add primitive GroupByKeyTranslatorBatch implementation
     new 57ce2d1  Use Iterators.transform() to return Iterable
     new 4f150da  Implement read transform
     new 1ec9356  update TODO
     new ebbab69  Apply spotless
     new d531bb5  start source instanciation
     new a3a87b4  Improve exception flow
     new b7283d7  Improve type enforcement in ReadSourceTranslator
     new e9ac3c3  Experiment over using spark Catalog to pass in Beam Source through spark Table
     new 0452733  Add source mocks
     new 8cdc20f  fix mock, wire mock in translators and create a main test.
     new 1060121  Use raw WindowedValue so that spark Encoders could work (temporary)
     new 1184022  clean deps
     new 49ee259  Move DatasetSourceMock to proper batch mode
     new 340991e  Run pipeline in batch mode or in streaming mode
     new 1ca4192  Split batch and streaming sources and translators
     new 4e0f7a0  Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
     new 758c1ce  Cleaning
     new 92a104e  Add ReadSourceTranslatorStreaming
     new 2f5bdd3  Move Source and translator mocks to a mock package.
     new 1cea29d  Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
     new 92c94b1  Refactor DatasetSource fields
     new d1b549e  Wire real SourceTransform and not mock and update the test
     new 878ff4e  Add missing 0-arg public constructor
     new 6392179  Apply spotless

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3533779)
            \
             N -- N -- N   refs/heads/spark-runner_structured-streaming (6392179)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 50 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/PULL_REQUEST_TEMPLATE.md                   |    4 +-
 .gitignore                                         |    1 +
 .test-infra/jenkins/CommonJobProperties.groovy     |    2 +-
 .../job_PerformanceTests_FileBasedIO_IT.groovy     |   16 +
 ...job_PerformanceTests_FileBasedIO_IT_HDFS.groovy |   13 +
 .../jenkins/job_PerformanceTests_Python.groovy     |    2 +-
 ...GradleBuild.groovy => job_PostCommit_Go.groovy} |    3 +-
 ...adleBuild.groovy => job_PostCommit_Java.groovy} |    3 +-
 .../job_PostCommit_Java_Nexmark_Flink.groovy       |   10 +-
 ...y => job_PostCommit_Java_PortabilityApi.groovy} |    3 +-
 ...job_PostCommit_Java_ValidatesRunner_Apex.groovy |    5 +-
 ...PostCommit_Java_ValidatesRunner_Dataflow.groovy |    5 +-
 ...nner_DataflowPortabilityExecutableStage.groovy} |   11 +-
 ...ob_PostCommit_Java_ValidatesRunner_Flink.groovy |    3 +-
 ...PostCommit_Java_ValidatesRunner_Gearpump.groovy |    5 +-
 ..._ValidatesRunner_PortabilityApi_Dataflow.groovy |    3 +-
 ...ob_PostCommit_Java_ValidatesRunner_Samza.groovy |    3 +-
 ...ob_PostCommit_Java_ValidatesRunner_Spark.groovy |    5 +-
 ...radleBuild.groovy => job_PostCommit_SQL.groovy} |   11 +-
 .../jenkins/job_PreCommit_Portable_Python.groovy   |   18 +-
 ...t.groovy => job_Release_NightlySnapshot.groovy} |    3 +-
 .../job_beam_PerformanceTests_Analysis.groovy      |    2 +
 .test-infra/metrics/OWNERS                         |    3 +
 .test-infra/metrics/beamgrafana-deploy.yaml        |    2 +-
 .../dashboards/stability_critical_jobs_status.json |    4 +-
 .test-infra/metrics/sync/jenkins/syncjenkins.py    |    3 +-
 README.md                                          |    6 +-
 build.gradle                                       |   10 +
 .../org/apache/beam/gradle/BeamModulePlugin.groovy |  281 ++-
 .../org/apache/beam/gradle/GrpcVendoring.groovy    |    8 +-
 .../org/apache/beam/gradle/VendorJavaPlugin.groovy |    4 +
 examples/java/build.gradle                         |    2 +-
 .../beam/examples/complete/TrafficRoutes.java      |    3 +-
 .../beam/examples/complete/game/UserScore.java     |   36 +-
 .../beam/examples/cookbook/FilterExamples.java     |    3 +-
 .../beam/examples/cookbook/TriggerExample.java     |    3 +-
 .../examples/complete/game/LeaderBoardTest.java    |    2 +-
 .../beam/examples/complete/game/UserScoreTest.java |   31 +-
 .../beam/examples/cookbook/JoinExamplesTest.java   |   38 +-
 .../beam/examples/cookbook/TriggerExampleTest.java |   23 +-
 model/fn-execution/build.gradle                    |    4 +-
 .../fn-execution/src/main/proto/beam_fn_api.proto  |  300 ++-
 model/job-management/build.gradle                  |    4 +-
 model/pipeline/build.gradle                        |    2 +-
 .../pipeline/src/main/proto/beam_runner_api.proto  |   40 +
 .../src/main/groovy/MobileGamingCommands.groovy    |   20 +-
 .../main/groovy/mobilegaming-java-dataflow.groovy  |  100 +-
 release/src/main/scripts/cut_release_branch.sh     |    4 +-
 .../operators/ApexProcessFnOperator.java           |   35 +-
 .../core/construction/ArtifactServiceStager.java   |    6 +-
 .../beam/runners/core/construction/BeamUrns.java   |    2 +-
 .../core/construction/CoderTranslation.java        |    2 +-
 .../core/construction/CombineTranslation.java      |    2 +-
 .../CreatePCollectionViewTranslation.java          |    2 +-
 .../core/construction/DisplayDataTranslation.java  |    4 +-
 .../runners/core/construction/Environments.java    |   31 +-
 .../construction/ExecutableStageTranslation.java   |   90 +
 .../construction/PCollectionViewTranslation.java   |    2 +-
 .../core/construction/PTransformTranslation.java   |    6 +
 .../core/construction/ParDoTranslation.java        |    4 +-
 ...java => PipelineOptionsSerializationUtils.java} |   34 +-
 .../construction/PipelineOptionsTranslation.java   |    6 +-
 .../runners/core/construction/ReadTranslation.java |    4 +-
 .../construction/SerializablePipelineOptions.java  |   26 +-
 .../runners/core/construction/SplittableParDo.java |    2 +
 .../construction/SplittableParDoNaiveBounded.java  |    4 +-
 .../core/construction/TestStreamTranslation.java   |    2 +-
 .../core/construction/WindowIntoTranslation.java   |    2 +-
 .../construction/WindowingStrategyTranslation.java |    8 +-
 .../core/construction/WriteFilesTranslation.java   |    2 +-
 .../graph/GreedyPCollectionFusers.java             |   30 +-
 .../core/construction/graph/QueryablePipeline.java |    8 +-
 .../construction/ArtifactServiceStagerTest.java    |    6 +-
 .../runners/core/construction/CommonCoderTest.java |    2 +-
 .../ExecutableStageTranslationTest.java            |  120 ++
 .../InMemoryArtifactStagerService.java             |    2 +-
 .../PipelineOptionsTranslationTest.java            |    6 +-
 .../construction/WindowIntoTranslationTest.java    |    2 +-
 .../construction/graph/ProtoOverridesTest.java     |    2 +-
 .../runners/core/LateDataDroppingDoFnRunner.java   |   17 +-
 .../apache/beam/runners/core/SideInputHandler.java |    3 +-
 .../core/SplittableParDoViaKeyedWorkItems.java     |    7 +-
 .../runners/core/metrics/DistributionCell.java     |    5 +
 .../runners/core/metrics/MetricsContainerImpl.java |   19 +
 .../beam/runners/core/metrics/MetricsPusher.java   |    6 +-
 .../beam/runners/core/metrics/NoOpMetricsSink.java |    4 +-
 .../core/metrics/SimpleMonitoringInfoBuilder.java  |  219 +++
 .../core/serialization}/Base64Serializer.java      |    4 +-
 .../runners/core/serialization}/package-info.java  |    4 +-
 .../beam/runners/core/ReduceFnRunnerTest.java      |    2 +-
 .../runners/core/SplittableParDoProcessFnTest.java |   20 +-
 .../runners/core/metrics/MetricsPusherTest.java    |    7 +-
 .../metrics/SimpleMonitoringInfoBuilderTest.java   |   87 +
 .../beam/runners/core/metrics/TestMetricsSink.java |    4 +-
 runners/direct-java/build.gradle                   |   35 +-
 .../runners/direct/ParDoMultiOverrideFactory.java  |   32 +-
 .../runners/direct/WindowEvaluatorFactory.java     |    3 +-
 .../runners/direct/portable/ReferenceRunner.java   |   15 +-
 .../direct/portable/WindowEvaluatorFactory.java    |    3 +-
 .../LocalFileSystemArtifactRetrievalService.java   |    6 +-
 .../LocalFileSystemArtifactStagerService.java      |    8 +-
 .../runners/direct/portable/job/PreparingJob.java  |    2 +-
 .../portable/job/ReferenceRunnerJobService.java    |    6 +-
 .../beam/runners/direct/DirectRunnerTest.java      |   12 +-
 .../direct/UnboundedReadEvaluatorFactoryTest.java  |    5 +-
 .../runners/direct/WindowEvaluatorFactoryTest.java |    3 +-
 .../direct/portable/ReferenceRunnerTest.java       |    3 +-
 .../portable/RemoteStageEvaluatorFactoryTest.java  |    5 +-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |    4 +-
 .../LocalFileSystemArtifactStagerServiceTest.java  |   10 +-
 .../UnsupportedArtifactRetrievalServiceTest.java   |    2 +-
 .../job/ReferenceRunnerJobServiceTest.java         |    4 +-
 runners/extensions-java/metrics/build.gradle       |    2 +-
 .../extensions/metrics/MetricsGraphiteSink.java    |    4 +-
 .../extensions/metrics/MetricsHttpSink.java        |    4 +-
 .../metrics/MetricsGraphiteSinkTest.java           |    6 +-
 .../extensions/metrics/MetricsHttpSinkTest.java    |    6 +-
 {model/pipeline => runners/flink/1.6}/build.gradle |   22 +-
 .../flink/1.6/job-server-container}/build.gradle   |   10 +-
 .../flink/1.6/job-server}/build.gradle             |   19 +-
 runners/flink/build.gradle                         |  129 +-
 .../flink/{build.gradle => flink_runner.gradle}    |   34 +-
 runners/flink/job-server-container/Dockerfile      |    6 +-
 runners/flink/job-server-container/build.gradle    |   38 +-
 ...ld.gradle => flink_job_server_container.gradle} |   26 +-
 runners/flink/job-server/build.gradle              |   80 +-
 .../{build.gradle => flink_job_server.gradle}      |   79 +-
 .../FlinkBatchPortablePipelineTranslator.java      |   71 +-
 .../runners/flink/FlinkExecutionEnvironments.java  |  173 +-
 .../beam/runners/flink/FlinkJobInvocation.java     |   88 +-
 .../apache/beam/runners/flink/FlinkJobInvoker.java |    3 +-
 .../beam/runners/flink/FlinkJobServerDriver.java   |   25 +-
 .../flink/FlinkPipelineExecutionEnvironment.java   |   15 +-
 .../beam/runners/flink/FlinkPipelineOptions.java   |   22 +
 .../flink/FlinkPortablePipelineTranslator.java     |   19 +-
 .../org/apache/beam/runners/flink/FlinkRunner.java |    5 +-
 .../flink/FlinkStreamingPipelineTranslator.java    |   70 +
 .../FlinkStreamingPortablePipelineTranslator.java  |   49 +-
 .../flink/FlinkStreamingTransformTranslators.java  |   26 +-
 .../runners/flink/FlinkTransformOverrides.java     |   24 +-
 ....java => PipelineTranslationModeOptimizer.java} |   26 +-
 .../beam/runners/flink/metrics/FileReporter.java   |   75 +
 .../flink/metrics/FlinkMetricContainer.java        |   97 +-
 .../apache/beam/runners/flink/metrics/Metrics.java |   56 +
 .../FlinkDefaultExecutableStageContext.java        |    3 +
 .../functions/FlinkExecutableStageFunction.java    |   23 +-
 .../utils/FlinkPipelineTranslatorUtils.java        |    2 +-
 .../runners/flink/translation/utils/NoopLock.java  |   72 +
 .../wrappers/streaming/DoFnOperator.java           |    3 +-
 .../streaming/ExecutableStageDoFnOperator.java     |  144 +-
 .../streaming/io/UnboundedSourceWrapper.java       |   46 +-
 .../state/FlinkKeyGroupStateInternals.java         |    3 +-
 .../streaming/state/FlinkStateInternals.java       |   56 +-
 .../flink/FlinkExecutionEnvironmentsTest.java      |  252 ++-
 .../runners/flink/FlinkJobServerDriverTest.java    |   27 +-
 .../FlinkPipelineExecutionEnvironmentTest.java     |   26 +
 .../FlinkStreamingTransformTranslatorsTest.java    |  238 +++
 .../runners/flink/FlinkTransformOverridesTest.java |  116 ++
 .../beam/runners/flink/PipelineOptionsTest.java    |    3 +
 .../PipelineTranslationModeOptimizerTest.java      |   63 +
 .../beam/runners/flink/PortableExecutionTest.java  |   95 +-
 .../runners/flink/PortableStateExecutionTest.java  |  194 +-
 .../runners/flink/PortableTimersExecutionTest.java |   36 +-
 .../flink/metrics/FlinkMetricContainerTest.java    |  134 ++
 .../flink/streaming/BoundedSourceRestoreTest.java  |    1 +
 .../streaming/ExecutableStageDoFnOperatorTest.java |    2 +-
 .../flink/streaming/GroupByWithNullValuesTest.java |   92 +
 .../FlinkPipelineTranslatorUtilsTest.java          |   44 +
 .../FlinkDefaultExecutableStageContextTest.java    |    2 +-
 .../FlinkExecutableStageFunctionTest.java          |    2 +-
 .../wrappers/streaming/io}/TestCountingSource.java |   17 +-
 .../streaming/io}/UnboundedSourceWrapperTest.java  |  150 +-
 runners/google-cloud-dataflow-java/build.gradle    |   64 +-
 .../examples-streaming/build.gradle                |    2 +-
 .../examples/build.gradle                          |    6 +-
 .../beam/runners/dataflow/DataflowPipelineJob.java |    6 +-
 .../dataflow/DataflowPipelineTranslator.java       |    2 +-
 .../beam/runners/dataflow/DataflowRunner.java      |    3 +
 .../dataflow/options/DataflowPipelineOptions.java  |    8 +
 .../runners/dataflow/util/CloudObjectKinds.java    |    1 +
 .../dataflow/util/CloudObjectTranslators.java      |    6 +-
 .../beam/runners/dataflow/util/GcsStager.java      |    4 +-
 .../beam/runners/dataflow/util/PackageUtil.java    |    9 +-
 .../beam/runners/dataflow/util/GCSUploadMain.java} |   29 +-
 .../google-cloud-dataflow-java/worker/build.gradle |   26 +-
 .../worker/legacy-worker/build.gradle              |  114 +-
 .../dataflow/worker/BatchDataflowWorker.java       |   59 +-
 .../worker/BeamFnMapTaskExecutorFactory.java       |  113 +-
 .../runners/dataflow/worker/ByteStringCoder.java   |    2 +-
 .../worker/DataflowMapTaskExecutorFactory.java     |   13 +-
 .../dataflow/worker/DataflowOperationContext.java  |    2 +-
 .../dataflow/worker/DataflowRunnerHarness.java     |   23 +-
 .../worker/DataflowWorkerHarnessHelper.java        |   15 +-
 .../dataflow/worker/DeltaDistributionCell.java     |    5 +
 .../runners/dataflow/worker/ExperimentContext.java |    6 +-
 ...FetchAndFilterStreamingSideInputsOperation.java |    6 +-
 .../dataflow/worker/FnApiWindowMappingFn.java      |   12 +-
 .../worker/GroupAlsoByWindowParDoFnFactory.java    |    2 +-
 .../worker/IntrinsicMapTaskExecutorFactory.java    |   15 +-
 .../dataflow/worker/IsmSideInputReader.java        |    3 +-
 .../worker/MetricTrackingWindmillServerStub.java   |    2 +-
 .../beam/runners/dataflow/worker/PubsubSink.java   |    2 +-
 .../beam/runners/dataflow/worker/ReaderCache.java  |    2 +-
 ...HarnessCoderCloudObjectTranslatorRegistrar.java |    2 -
 .../dataflow/worker/SdkHarnessRegistries.java      |   16 +-
 .../dataflow/worker/SdkHarnessRegistry.java        |    9 +-
 .../beam/runners/dataflow/worker/StateFetcher.java |    2 +-
 .../dataflow/worker/StreamingDataflowWorker.java   |   53 +-
 .../worker/StreamingModeExecutionContext.java      |   30 +-
 .../dataflow/worker/StreamingSideInputFetcher.java |    4 +-
 .../dataflow/worker/WindmillNamespacePrefix.java   |    2 +-
 .../beam/runners/dataflow/worker/WindmillSink.java |    2 +-
 .../dataflow/worker/WindmillStateCache.java        |    2 +-
 .../dataflow/worker/WindmillStateInternals.java    |    2 +-
 .../dataflow/worker/WindmillStateReader.java       |    2 +-
 .../dataflow/worker/WindmillTimerInternals.java    |    2 +-
 .../dataflow/worker/WorkItemStatusClient.java      |   27 +-
 .../dataflow/worker/WorkerCustomSources.java       |    2 +-
 .../FixMultiOutputInfosOnParDoInstructions.java    |    8 +-
 .../dataflow/worker/fn/BeamFnControlService.java   |    2 +-
 .../runners/dataflow/worker/fn/ServerFactory.java  |  229 ---
 .../dataflow/worker/fn/SocketAddressFactory.java   |   68 -
 .../worker/fn/control/BeamFnMapTaskExecutor.java   |  262 ++-
 .../fn/control/ProcessRemoteBundleOperation.java   |  105 +
 .../control/RegisterAndProcessBundleOperation.java |   38 +-
 .../worker/fn/data/BeamFnDataGrpcService.java      |   14 +-
 .../fn/data/RemoteGrpcPortReadOperation.java       |    8 +-
 .../fn/data/RemoteGrpcPortWriteOperation.java      |    9 +-
 .../worker/fn/logging/BeamFnLoggingService.java    |    4 +-
 .../fn/stream/ServerStreamObserverFactory.java     |    6 +-
 .../graph/CloneAmbiguousFlattensFunction.java      |    9 +-
 ...java => CreateExecutableStageNodeFunction.java} |  345 +---
 .../graph/CreateRegisterFnOperationFunction.java   |   49 +-
 ...nsertFetchAndFilterStreamingSideInputNodes.java |    7 +-
 .../worker/graph/LengthPrefixUnknownCoders.java    |    7 +-
 .../worker/graph/MapTaskToNetworkFunction.java     |   11 +-
 .../beam/runners/dataflow/worker/graph/Nodes.java  |   43 +-
 .../worker/graph/RegisterNodeFunction.java         |   30 +-
 .../common/worker/BatchingShuffleEntryReader.java  |    3 +-
 .../worker/util/common/worker/MapTaskExecutor.java |    2 +-
 .../worker/windmill/DirectStreamObserver.java      |    4 +-
 .../windmill/ForwardingClientResponseObserver.java |    6 +-
 .../worker/windmill/GrpcWindmillServer.java        |   32 +-
 .../worker/windmill/StreamObserverFactory.java     |    4 +-
 .../runners/dataflow/harness/test/TestStreams.java |    4 +-
 .../runners/dataflow/worker/ConcatReaderTest.java  |    3 +-
 .../runners/dataflow/worker/DataflowMatchers.java  |    2 +-
 .../worker/DataflowWorkerHarnessHelperTest.java    |    2 +-
 .../dataflow/worker/FnApiWindowMappingFnTest.java  |    4 +-
 .../dataflow/worker/GroupingShuffleReaderTest.java |    2 +-
 .../IntrinsicMapTaskExecutorFactoryTest.java       |   37 +-
 .../dataflow/worker/IsmSideInputReaderTest.java    |    3 +-
 .../runners/dataflow/worker/PubsubReaderTest.java  |    2 +-
 .../runners/dataflow/worker/PubsubSinkTest.java    |    2 +-
 .../runners/dataflow/worker/ReaderCacheTest.java   |    2 +-
 .../dataflow/worker/ShuffleReaderFactoryTest.java  |    3 +-
 .../runners/dataflow/worker/StateFetcherTest.java  |    2 +-
 .../worker/StreamingDataflowWorkerTest.java        |    6 +-
 .../worker/StreamingGroupAlsoByWindowFnsTest.java  |    2 +-
 ...reamingGroupAlsoByWindowsReshuffleDoFnTest.java |    2 +-
 .../worker/StreamingModeExecutionContextTest.java  |   16 +-
 .../worker/StreamingSideInputDoFnRunnerTest.java   |    2 +-
 .../worker/StreamingSideInputFetcherTest.java      |    2 +-
 .../dataflow/worker/WindmillKeyedWorkItemTest.java |    2 +-
 .../worker/WindmillReaderIteratorBaseTest.java     |    2 +-
 .../dataflow/worker/WindmillStateCacheTest.java    |    2 +-
 .../worker/WindmillStateInternalsTest.java         |    2 +-
 .../dataflow/worker/WindmillStateReaderTest.java   |    4 +-
 .../dataflow/worker/WorkerCustomSourcesTest.java   |    4 +-
 ...FixMultiOutputInfosOnParDoInstructionsTest.java |   20 +-
 .../worker/fn/BeamFnControlServiceTest.java        |   13 +-
 .../dataflow/worker/fn/ServerFactoryTest.java      |  244 ---
 .../worker/fn/SocketAddressFactoryTest.java        |   55 -
 .../fn/control/BeamFnMapTaskExecutorTest.java      |  219 ++-
 .../RegisterAndProcessBundleOperationTest.java     |   96 +-
 .../SingularProcessBundleProgressTrackerTest.java  |    6 +-
 .../worker/fn/data/BeamFnDataGrpcServiceTest.java  |   32 +-
 .../fn/data/RemoteGrpcPortReadOperationTest.java   |   14 +-
 .../fn/data/RemoteGrpcPortWriteOperationTest.java  |   16 +-
 .../fn/logging/BeamFnLoggingServiceTest.java       |   12 +-
 .../fn/stream/ServerStreamObserverFactoryTest.java |    4 +-
 .../graph/CloneAmbiguousFlattensFunctionTest.java  |    7 +-
 .../CreateRegisterFnOperationFunctionTest.java     |    6 +-
 .../graph/DeduceFlattenLocationsFunctionTest.java  |    2 +-
 .../graph/DeduceNodeLocationsFunctionTest.java     |    4 +-
 ...tFetchAndFilterStreamingSideInputNodesTest.java |    9 +-
 .../graph/LengthPrefixUnknownCodersTest.java       |    2 +-
 .../worker/graph/MapTaskToNetworkFunctionTest.java |   25 +-
 .../runners/dataflow/worker/graph/NodesTest.java   |   16 +-
 .../RemoveFlattenInstructionsFunctionTest.java     |   74 +-
 .../ReplacePgbkWithPrecombineFunctionTest.java     |    2 +-
 .../logging/DataflowWorkerLoggingHandlerTest.java  |    2 +-
 .../util/common/worker/ReadOperationTest.java      |    6 +-
 .../worker/windmill/GrpcWindmillServerTest.java    |   10 +-
 .../worker/windmill/build.gradle                   |    2 +-
 .../apache/beam/runners/fnexecution/FnService.java |    6 +-
 .../GrpcContextHeaderAccessorProvider.java         |   16 +-
 .../beam/runners/fnexecution/GrpcFnServer.java     |   19 +-
 .../fnexecution/InProcessServerFactory.java        |   39 +-
 .../beam/runners/fnexecution/ServerFactory.java    |  192 +-
 .../BeamFileSystemArtifactRetrievalService.java    |   10 +-
 .../BeamFileSystemArtifactStagingService.java      |   10 +-
 .../control/DefaultJobBundleFactory.java           |    4 +-
 .../fnexecution/control/FnApiControlClient.java    |    6 +-
 .../control/FnApiControlClientPoolService.java     |    2 +-
 .../control/ProcessBundleDescriptors.java          |    2 +-
 .../SingleEnvironmentInstanceJobBundleFactory.java |   13 +-
 .../runners/fnexecution/data/GrpcDataService.java  |   11 +-
 ...actory.java => ExternalEnvironmentFactory.java} |  116 +-
 .../environment/ProcessEnvironmentFactory.java     |    2 +-
 .../environment/StaticRemoteEnvironment.java       |   64 +
 .../StaticRemoteEnvironmentFactory.java            |   70 +
 .../jobsubmission/InMemoryJobService.java          |   10 +-
 .../fnexecution/jobsubmission/JobInvoker.java      |    2 +-
 .../fnexecution/jobsubmission/JobPreparation.java  |    2 +-
 .../fnexecution/logging/GrpcLoggingService.java    |    2 +-
 .../runners/fnexecution/provisioning/JobInfo.java  |    2 +-
 .../provisioning/StaticGrpcProvisionService.java   |    2 +-
 .../splittabledofn/SDFFeederViaStateAndTimers.java |    4 +-
 .../fnexecution/state/GrpcStateService.java        |    6 +-
 .../fnexecution/state/StateRequestHandlers.java    |    2 +-
 .../GrpcContextHeaderAccessorProviderTest.java     |   24 +-
 .../runners/fnexecution/ServerFactoryTest.java     |   48 +-
 .../BeamFileSystemArtifactServicesTest.java        |    8 +-
 .../control/DefaultJobBundleFactoryTest.java       |    4 +-
 .../control/FnApiControlClientPoolServiceTest.java |    4 +-
 .../control/FnApiControlClientTest.java            |    2 +-
 .../fnexecution/control/RemoteExecutionTest.java   |  220 ++-
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |    3 +-
 .../fnexecution/data/GrpcDataServiceTest.java      |    8 +-
 .../jobsubmission/InMemoryJobServiceTest.java      |    4 +-
 .../logging/GrpcLoggingServiceTest.java            |    6 +-
 .../StaticGrpcProvisionServiceTest.java            |   10 +-
 .../fnexecution/state/GrpcStateServiceTest.java    |    4 +-
 .../apache/beam/runners/local/StructuralKey.java   |   14 +-
 runners/reference/java/build.gradle                |    2 +
 .../runners/reference/ExternalWorkerService.java   |   87 +
 .../reference/JobServicePipelineResult.java        |   10 +-
 .../beam/runners/reference/PortableRunner.java     |   34 +-
 .../runners/reference/testing/TestJobService.java  |    2 +-
 .../beam/runners/reference/PortableRunnerTest.java |    4 +-
 runners/reference/job-server/build.gradle          |    1 +
 runners/samza/build.gradle                         |    2 +-
 .../org/apache/beam/runners/samza/SamzaRunner.java |    2 +-
 .../runners/samza/adapter/BoundedSourceSystem.java |    2 +-
 .../samza/adapter/UnboundedSourceSystem.java       |    2 +-
 .../apache/beam/runners/samza/runtime/DoFnOp.java  |   14 +-
 .../beam/runners/samza/runtime/GroupByKeyOp.java   |    2 +-
 .../samza/runtime/SamzaDoFnInvokerRegistrar.java   |   21 +-
 .../samza/runtime/SamzaStoreStateInternals.java    |   10 +-
 .../samza/transforms/GroupWithoutRepartition.java  |   60 +
 .../samza/transforms/UpdatingCombineFn.java        |   19 +-
 .../runners/samza/transforms}/package-info.java    |    4 +-
 .../runners/samza/translation/ConfigBuilder.java   |    2 +-
 .../runners/samza/translation/ConfigContext.java   |    9 +-
 .../samza/translation/GroupByKeyTranslator.java    |   22 +-
 .../translation/ParDoBoundMultiTranslator.java     |   17 +
 .../runners/samza/translation/ReadTranslator.java  |    2 +-
 .../samza/translation/SamzaPipelineTranslator.java |    7 +-
 .../samza/translation/TranslationContext.java      |   47 +-
 .../runners/samza/util/PipelineDotRenderer.java    |    3 +-
 .../translation/TranslationContext.java            |   10 +-
 .../translation/batch/DatasetSourceBatch.java      |   62 +-
 .../translation/batch/FlattenTranslatorBatch.java  |    2 +-
 .../translation/batch/PipelineTranslatorBatch.java |    2 +-
 .../batch/ReadSourceTranslatorBatch.java           |   34 +-
 .../batch/{ => mocks}/DatasetSourceMockBatch.java  |   46 +-
 .../{ => mocks}/ReadSourceTranslatorMockBatch.java |   22 +-
 .../streaming/DatasetSourceStreaming.java          |   16 +-
 .../streaming/ReadSourceTranslatorStreaming.java   |   22 +-
 .../spark/structuredstreaming/SourceTest.java      |   21 +-
 runners/spark/build.gradle                         |    6 +-
 .../org/apache/beam/runners/spark/SparkRunner.java |    4 +-
 .../beam/runners/spark/examples/WordCount.java     |    4 +-
 .../spark/translation/TranslationUtils.java        |    5 +-
 .../streaming/StreamingTransformTranslator.java    |    4 +-
 .../runners/spark/ProvidedSparkContextTest.java    |    2 +-
 .../spark/metrics/SparkMetricsPusherTest.java      |   10 +-
 sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go    |   14 +-
 sdks/go/pkg/beam/artifact/gcsproxy/staging.go      |    6 +-
 sdks/go/pkg/beam/core/graph/coder/int.go           |   24 +-
 sdks/go/pkg/beam/core/graph/coder/varint.go        |   10 +-
 sdks/go/pkg/beam/core/graph/fn.go                  |   11 +
 sdks/go/pkg/beam/core/runtime/exec/coder.go        |    3 +-
 sdks/go/pkg/beam/core/runtime/exec/fn.go           |  104 +-
 sdks/go/pkg/beam/core/runtime/exec/fn_arity.go     |  251 +++
 sdks/go/pkg/beam/core/runtime/exec/fn_arity.tmpl   |   69 +
 sdks/go/pkg/beam/core/runtime/exec/fn_test.go      |  242 ++-
 sdks/go/pkg/beam/core/runtime/exec/fullvalue.go    |   58 +
 .../pkg/beam/core/runtime/exec/fullvalue_test.go   |   58 +
 sdks/go/pkg/beam/core/runtime/init.go              |    2 +-
 sdks/go/pkg/beam/core/util/ioutilx/read.go         |   42 +
 .../beam/core/util/ioutilx/{read.go => write.go}   |   30 +-
 sdks/go/pkg/beam/core/util/reflectx/call.go        |    2 +-
 sdks/go/pkg/beam/core/util/reflectx/structs.go     |   73 +
 sdks/go/pkg/beam/forward.go                        |    4 +-
 sdks/go/pkg/beam/io/filesystem/gcs/gcs.go          |   70 +-
 sdks/go/pkg/beam/runners/dataflow/dataflow.go      |    4 +-
 .../pkg/beam/runners/dataflow/dataflowlib/stage.go |    4 +-
 .../beam/runners/universal/runnerlib/compile.go    |    2 +-
 sdks/go/pkg/beam/testing/ptest/ptest.go            |   43 +-
 sdks/go/pkg/beam/util/gcsx/gcs.go                  |   66 +-
 .../util/ioutilx/read.go => util/gcsx/gcs_test.go} |   40 +-
 sdks/go/pkg/beam/util/shimx/generate.go            |   49 +-
 sdks/go/pkg/beam/util/shimx/generate_test.go       |   17 +
 sdks/go/pkg/beam/util/starcgenx/starcgenx.go       |   69 +-
 sdks/go/pkg/beam/util/starcgenx/starcgenx_test.go  |    4 +-
 sdks/go/pkg/beam/x/hooks/perf/perf.go              |   68 +-
 sdks/java/bom/build.gradle                         |  122 ++
 sdks/java/bom/pom.xml.template                     |   83 +
 sdks/java/build-tools/build.gradle                 |    2 +-
 .../src/main/resources/beam/checkstyle.xml         |   10 +
 .../java/org/apache/beam/sdk/coders/ListCoder.java |   19 +
 .../java/org/apache/beam/sdk/coders/MapCoder.java  |   19 +
 .../apache/beam/sdk/coders/RowCoderGenerator.java  |    4 +-
 .../main/java/org/apache/beam/sdk/io/AvroIO.java   |   81 +-
 .../org/apache/beam/sdk/io/BlockBasedSource.java   |    4 +-
 .../org/apache/beam/sdk/io/LocalFileSystem.java    |    6 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java    |    6 +-
 .../main/java/org/apache/beam/sdk/io/TextIO.java   |    2 +-
 .../org/apache/beam/sdk/metrics/Distribution.java  |    2 +
 .../java/org/apache/beam/sdk/metrics/Metrics.java  |    8 +
 .../apache/beam/sdk/metrics/MetricsOptions.java    |   83 +
 .../apache/beam/sdk/options/PipelineOptions.java   |   55 -
 .../org/apache/beam/sdk/options/ValueProvider.java |    6 +-
 .../apache/beam/sdk/schemas/AvroRecordSchema.java  |   53 +
 .../apache/beam/sdk/schemas/CachingFactory.java    |   55 +
 .../java/org/apache/beam/sdk/schemas/Factory.java} |    9 +-
 .../beam/sdk/schemas/FieldTypeDescriptors.java     |    3 +-
 .../apache/beam/sdk/schemas/FieldValueGetter.java  |    2 -
 .../beam/sdk/schemas/FieldValueGetterFactory.java  |    6 +-
 .../apache/beam/sdk/schemas/FieldValueSetter.java  |   16 -
 .../beam/sdk/schemas/FieldValueSetterFactory.java  |    6 +-
 .../sdk/schemas/FieldValueTypeInformation.java     |  224 +++
 .../schemas/FieldValueTypeInformationFactory.java} |   12 +-
 .../beam/sdk/schemas/FromRowUsingCreator.java      |  149 ++
 .../sdk/schemas/GetterBasedSchemaProvider.java     |  175 +-
 .../apache/beam/sdk/schemas/JavaBeanSchema.java    |   70 +-
 .../apache/beam/sdk/schemas/JavaFieldSchema.java   |   42 +-
 .../java/org/apache/beam/sdk/schemas/Schema.java   |  107 +-
 .../schemas/SchemaUserTypeConstructorCreator.java  |   44 +
 .../beam/sdk/schemas/SchemaUserTypeCreator.java}   |   13 +-
 .../sdk/schemas/SetterBasedCreatorFactory.java     |   57 +
 .../beam/sdk/schemas/UserTypeCreatorFactory.java   |   14 +-
 .../apache/beam/sdk/schemas/transforms/Cast.java   |    2 +-
 .../apache/beam/sdk/schemas/transforms/Select.java |    7 +-
 .../beam/sdk/schemas/utils/AvroByteBuddyUtils.java |  125 ++
 .../apache/beam/sdk/schemas/utils/AvroUtils.java   |  686 ++++++-
 .../beam/sdk/schemas/utils/ByteBuddyUtils.java     |  159 +-
 ...terFactory.java => FieldValueTypeSupplier.java} |   20 +-
 .../sdk/schemas/utils/JavaBeanGetterFactory.java   |   31 -
 .../sdk/schemas/utils/JavaBeanSetterFactory.java   |   31 -
 .../beam/sdk/schemas/utils/JavaBeanUtils.java      |  151 +-
 .../apache/beam/sdk/schemas/utils/POJOUtils.java   |  211 +-
 .../sdk/schemas/utils/PojoValueSetterFactory.java  |   31 -
 .../beam/sdk/schemas/utils/ReflectUtils.java       |   20 +-
 .../sdk/schemas/utils/StaticSchemaInference.java   |  100 +-
 ...aflowPortabilityExecutableStageUnsupported.java |   17 +-
 .../apache/beam/sdk/testing/UsesSideInputs.java    |   16 +-
 .../org/apache/beam/sdk/transforms/Contextful.java |    4 +-
 .../org/apache/beam/sdk/transforms/Create.java     |    9 +-
 .../java/org/apache/beam/sdk/transforms/DoFn.java  |   49 +-
 .../org/apache/beam/sdk/transforms/Filter.java     |   23 +-
 .../beam/sdk/transforms/FlatMapElements.java       |   40 +-
 ...{SimpleFunction.java => InferableFunction.java} |   49 +-
 .../apache/beam/sdk/transforms/MapElements.java    |   36 +-
 ...ializableFunction.java => ProcessFunction.java} |   17 +-
 .../beam/sdk/transforms/SerializableFunction.java  |   11 +-
 .../apache/beam/sdk/transforms/SimpleFunction.java |   38 +-
 .../org/apache/beam/sdk/transforms/ToString.java   |    8 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java       |    6 +-
 .../reflect/ByteBuddyOnTimerInvokerFactory.java    |    4 +-
 .../beam/sdk/transforms/reflect/DoFnInvoker.java   |    2 +
 .../sdk/transforms/reflect/DoFnSignatures.java     |   13 +-
 .../sdk/transforms/splittabledofn/Backlog.java     |   90 +
 .../sdk/transforms/splittabledofn/Backlogs.java    |   58 +
 .../splittabledofn/ByteKeyRangeTracker.java        |   30 +-
 .../splittabledofn/OffsetRangeTracker.java         |   28 +-
 .../splittabledofn/RestrictionTracker.java         |    9 +-
 .../transforms/splittabledofn/Restrictions.java    |   17 +-
 .../java/org/apache/beam/sdk/util/CoderUtils.java  |    3 +-
 .../main/java/org/apache/beam/sdk/values/Row.java  |   40 +-
 .../org/apache/beam/sdk/values/RowWithGetters.java |   13 +-
 .../apache/beam/sdk/values/TypeDescriptors.java    |   68 +-
 .../avro/org/apache/beam/sdk/schemas/test.avsc     |   29 +
 .../org/apache/beam/sdk/coders/ListCoderTest.java  |   21 +
 .../org/apache/beam/sdk/coders/MapCoderTest.java   |   21 +
 .../java/org/apache/beam/sdk/io/AvroIOTest.java    | 2066 ++++++++++----------
 .../sdk/io/BoundedReadFromUnboundedSourceTest.java |    4 +-
 .../org/apache/beam/sdk/io/CountingSourceTest.java |   13 +-
 .../java/org/apache/beam/sdk/io/FileIOTest.java    |    2 +
 .../sdk/io/SerializableAvroCodecFactoryTest.java   |    2 +-
 .../org/apache/beam/sdk/io/TFRecordIOTest.java     |   11 +-
 .../org/apache/beam/sdk/io/TextIOReadTest.java     |    2 +
 .../apache/beam/sdk/schemas/AvroSchemaTest.java    |  368 ++++
 .../beam/sdk/schemas/FieldTypeDescriptorsTest.java |    4 +-
 .../beam/sdk/schemas/transforms/CastTest.java      |   37 +-
 .../beam/sdk/schemas/utils/AvroUtilsTest.java      |  335 +++-
 .../beam/sdk/schemas/utils/JavaBeanUtilsTest.java  |   24 +-
 .../beam/sdk/schemas/utils/POJOUtilsTest.java      |   20 +-
 .../org/apache/beam/sdk/testing/PAssertTest.java   |   16 +-
 .../apache/beam/sdk/transforms/CombineFnsTest.java |    5 +-
 .../apache/beam/sdk/transforms/CombineTest.java    |   70 +-
 .../org/apache/beam/sdk/transforms/FilterTest.java |   17 +
 .../beam/sdk/transforms/FlatMapElementsTest.java   |   60 +-
 .../apache/beam/sdk/transforms/FlattenTest.java    |    3 +-
 .../apache/beam/sdk/transforms/GroupByKeyTest.java |   11 +-
 .../beam/sdk/transforms/MapElementsTest.java       |  163 +-
 .../org/apache/beam/sdk/transforms/ParDoTest.java  |   17 +-
 .../org/apache/beam/sdk/transforms/ReifyTest.java  |    4 +-
 .../beam/sdk/transforms/ReifyTimestampsTest.java   |    4 +-
 .../apache/beam/sdk/transforms/ReshuffleTest.java  |   11 +-
 .../beam/sdk/transforms/SplittableDoFnTest.java    |   22 +-
 .../org/apache/beam/sdk/transforms/ViewTest.java   |    2 +
 .../org/apache/beam/sdk/transforms/WatchTest.java  |    9 +
 .../beam/sdk/transforms/join/CoGroupByKeyTest.java |    8 +-
 .../sdk/transforms/reflect/DoFnInvokersTest.java   |   20 +-
 .../reflect/DoFnSignaturesSplittableDoFnTest.java  |   52 +-
 .../splittabledofn/ByteKeyRangeTrackerTest.java    |   40 +
 .../splittabledofn/OffsetRangeTrackerTest.java     |   35 +-
 .../beam/sdk/transforms/windowing/WindowTest.java  |    5 +-
 .../sdk/transforms/windowing/WindowingTest.java    |    9 +-
 sdks/java/extensions/euphoria/build.gradle         |    7 +-
 .../core/translate/BeamMetricsTranslationTest.java |   14 +-
 .../sdk/extensions/gcp/options/GcsOptions.java     |   10 +
 .../sdk/extensions/gcp/storage/GcsFileSystem.java  |   20 +
 .../org/apache/beam/sdk/util/CustomHttpErrors.java |  141 ++
 .../java/org/apache/beam/sdk/util/GcsUtil.java     |   31 +-
 .../apache/beam/sdk/util/HttpCallCustomError.java  |   13 +-
 .../org/apache/beam/sdk/util/HttpCallMatcher.java  |   16 +-
 .../apache/beam/sdk/util/HttpRequestWrapper.java}  |   28 +-
 .../apache/beam/sdk/util/HttpResponseWrapper.java  |   24 +-
 .../beam/sdk/util/RetryHttpRequestInitializer.java |   32 +-
 .../apache/beam/sdk/util/CustomHttpErrorsTest.java |  128 ++
 .../apache/beam/sdk/util/gcsfs/GcsPathTest.java    |    3 +-
 sdks/java/extensions/kryo/build.gradle             |    7 +-
 .../beam/sdk/extensions/sorter/SortValues.java     |    2 +-
 sdks/java/extensions/sql/build.gradle              |    4 +-
 sdks/java/extensions/sql/jdbc/build.gradle         |    7 +-
 .../extensions/sql/src/main/codegen/config.fmpp    |  291 +++
 .../beam/sdk/extensions/sql/BeamSqlTable.java      |    3 +
 .../sdk/extensions/sql/impl/BeamCalciteTable.java  |    9 +-
 .../impl/{UdfImpl.java => ScalarFunctionImpl.java} |   34 +-
 .../beam/sdk/extensions/sql/impl/UdfImpl.java      |  144 +-
 .../interpreter/BeamSqlExpressionEnvironment.java  |   42 -
 .../interpreter/BeamSqlExpressionEnvironments.java |  147 --
 .../sql/impl/interpreter/BeamSqlFnExecutor.java    |  550 ------
 .../operator/BeamSqlBinaryOperator.java            |   40 -
 .../operator/BeamSqlCaseExpression.java            |   64 -
 .../operator/BeamSqlCastExpression.java            |  138 --
 .../operator/BeamSqlCorrelVariableExpression.java  |   48 -
 .../operator/BeamSqlDefaultExpression.java         |   38 -
 .../interpreter/operator/BeamSqlDotExpression.java |   57 -
 .../interpreter/operator/BeamSqlExpression.java    |   79 -
 .../operator/BeamSqlInputRefExpression.java        |   48 -
 .../operator/BeamSqlLocalRefExpression.java        |   48 -
 .../operator/BeamSqlOperatorExpression.java        |   52 -
 .../interpreter/operator/BeamSqlPrimitive.java     |  180 --
 .../interpreter/operator/BeamSqlUdfExpression.java |   89 -
 .../interpreter/operator/BeamSqlUnaryOperator.java |   41 -
 .../impl/interpreter/operator/DateOperators.java   |  164 --
 .../impl/interpreter/operator/StringOperators.java |  245 ---
 .../arithmetic/BeamSqlArithmeticExpression.java    |  125 --
 .../arithmetic/BeamSqlDivideExpression.java        |   35 -
 .../arithmetic/BeamSqlMinusExpression.java         |   34 -
 .../operator/arithmetic/BeamSqlModExpression.java  |   34 -
 .../arithmetic/BeamSqlMultiplyExpression.java      |   34 -
 .../operator/arithmetic/BeamSqlPlusExpression.java |   34 -
 .../operator/array/BeamSqlArrayExpression.java     |   51 -
 .../operator/array/BeamSqlArrayItemExpression.java |   50 -
 .../collection/BeamSqlCardinalityExpression.java   |   50 -
 .../collection/BeamSqlSingleElementExpression.java |   64 -
 .../comparison/BeamSqlCompareExpression.java       |   97 -
 .../comparison/BeamSqlEqualsExpression.java        |   53 -
 .../comparison/BeamSqlGreaterThanExpression.java   |   53 -
 .../BeamSqlGreaterThanOrEqualsExpression.java      |   53 -
 .../comparison/BeamSqlIsNotNullExpression.java     |   52 -
 .../comparison/BeamSqlIsNullExpression.java        |   52 -
 .../comparison/BeamSqlLessThanExpression.java      |   53 -
 .../BeamSqlLessThanOrEqualsExpression.java         |   53 -
 .../operator/comparison/BeamSqlLikeExpression.java |   51 -
 .../comparison/BeamSqlNotEqualsExpression.java     |   53 -
 .../comparison/BeamSqlNotLikeExpression.java       |   52 -
 .../date/BeamSqlCurrentDateExpression.java         |   49 -
 .../date/BeamSqlCurrentTimeExpression.java         |   53 -
 .../date/BeamSqlCurrentTimestampExpression.java    |   53 -
 .../date/BeamSqlDatetimeMinusExpression.java       |   98 -
 .../BeamSqlDatetimeMinusIntervalExpression.java    |   77 -
 .../date/BeamSqlDatetimePlusExpression.java        |  118 --
 .../date/BeamSqlIntervalMultiplyExpression.java    |   95 -
 .../BeamSqlTimestampMinusIntervalExpression.java   |   79 -
 .../BeamSqlTimestampMinusTimestampExpression.java  |   97 -
 .../interpreter/operator/date/TimeUnitUtils.java   |   63 -
 .../operator/logical/BeamSqlAndExpression.java     |   47 -
 .../operator/logical/BeamSqlNotExpression.java     |   48 -
 .../operator/logical/BeamSqlOrExpression.java      |   48 -
 .../operator/map/BeamSqlMapExpression.java         |   60 -
 .../operator/map/BeamSqlMapItemExpression.java     |   49 -
 .../operator/math/BeamSqlAbsExpression.java        |   63 -
 .../operator/math/BeamSqlAcosExpression.java       |   38 -
 .../operator/math/BeamSqlAsinExpression.java       |   38 -
 .../operator/math/BeamSqlAtan2Expression.java      |   41 -
 .../operator/math/BeamSqlAtanExpression.java       |   38 -
 .../operator/math/BeamSqlCeilExpression.java       |   43 -
 .../operator/math/BeamSqlCosExpression.java        |   38 -
 .../operator/math/BeamSqlCotExpression.java        |   38 -
 .../operator/math/BeamSqlDegreesExpression.java    |   38 -
 .../operator/math/BeamSqlExpExpression.java        |   38 -
 .../operator/math/BeamSqlFloorExpression.java      |   43 -
 .../operator/math/BeamSqlLnExpression.java         |   38 -
 .../operator/math/BeamSqlLogExpression.java        |   38 -
 .../operator/math/BeamSqlMathBinaryExpression.java |   63 -
 .../operator/math/BeamSqlMathUnaryExpression.java  |   57 -
 .../operator/math/BeamSqlPiExpression.java         |   44 -
 .../operator/math/BeamSqlPowerExpression.java      |   41 -
 .../operator/math/BeamSqlRadiansExpression.java    |   38 -
 .../operator/math/BeamSqlRandExpression.java       |   54 -
 .../math/BeamSqlRandIntegerExpression.java         |   58 -
 .../operator/math/BeamSqlRoundExpression.java      |  116 --
 .../operator/math/BeamSqlSignExpression.java       |   78 -
 .../operator/math/BeamSqlSinExpression.java        |   38 -
 .../operator/math/BeamSqlTanExpression.java        |   38 -
 .../operator/math/BeamSqlTruncateExpression.java   |   88 -
 .../interpreter/operator/math/package-info.java    |   20 -
 .../impl/interpreter/operator/package-info.java    |   20 -
 .../reinterpret/BeamSqlReinterpretExpression.java  |   70 -
 .../DatetimeReinterpretConversions.java            |   48 -
 .../reinterpret/IntegerReinterpretConversions.java |   36 -
 .../reinterpret/ReinterpretConversion.java         |  112 --
 .../operator/reinterpret/Reinterpreter.java        |   94 -
 .../operator/reinterpret/package-info.java         |   20 -
 .../operator/row/BeamSqlFieldAccessExpression.java |   69 -
 .../interpreter/operator/row/package-info.java     |   24 -
 .../sql/impl/interpreter/package-info.java         |   20 -
 .../sql/impl/parser/SqlCreateExternalTable.java    |    2 +-
 .../BeamJavaTypeFactory.java}                      |   33 +-
 .../sql/impl/planner/BeamRelDataTypeSystem.java    |    6 +
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |  391 +++-
 .../extensions/sql/impl/rel/BeamIOSourceRel.java   |    5 +
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  257 ++-
 .../sdk/extensions/sql/impl/rel/BeamRelNode.java   |   17 +
 .../sql/impl/rel/BeamSetOperatorRelBase.java       |   10 +-
 .../sql/impl/schema/BeamPCollectionTable.java      |    5 +
 .../extensions/sql/impl/schema/BeamTableUtils.java |    2 +-
 .../sql/impl/transform/BeamJoinTransforms.java     |   22 +-
 .../impl/transform/agg/CovarianceAccumulator.java  |   37 +-
 .../sql/impl/udf/BuiltinStringFunctions.java       |    8 +-
 .../extensions/sql/impl/utils/CalciteUtils.java    |    4 +-
 .../extensions/sql/impl/utils/SqlTypeUtils.java    |   59 -
 .../meta/provider/bigquery/BeamBigQueryTable.java  |    5 +
 .../sql/meta/provider/kafka/BeamKafkaTable.java    |    5 +
 .../meta/provider/kafka/KafkaTableProvider.java    |    3 +-
 .../meta/provider/pubsub/PubsubIOJsonTable.java    |    5 +
 .../provider/pubsub/PubsubJsonTableProvider.java   |    7 +-
 .../sql/meta/provider/test/TestBoundedTable.java   |    5 +
 .../sql/meta/provider/test/TestTableProvider.java  |    8 +-
 .../sql/meta/provider/test/TestUnboundedTable.java |    5 +
 .../sql/meta/provider/text/TextTable.java          |    5 +
 .../sql/meta/provider/text/TextTableProvider.java  |    5 +-
 .../beam/sdk/extensions/sql/BeamSqlCastTest.java   |   33 +-
 .../sql/BeamSqlDslSqlStdOperatorsTest.java         |   41 +-
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |   43 +-
 .../impl/interpreter/BeamSqlFnExecutorTest.java    |  204 --
 .../interpreter/BeamSqlFnExecutorTestBase.java     |   91 -
 .../operator/BeamNullExpressionTest.java           |   56 -
 .../operator/BeamSqlAndOrExpressionTest.java       |   70 -
 .../operator/BeamSqlCaseExpressionTest.java        |  101 -
 .../operator/BeamSqlCastExpressionTest.java        |  156 --
 .../operator/BeamSqlCompareExpressionTest.java     |  170 --
 .../operator/BeamSqlDotExpressionTest.java         |   76 -
 .../operator/BeamSqlInputRefExpressionTest.java    |   63 -
 .../interpreter/operator/BeamSqlPrimitiveTest.java |   95 -
 .../operator/BeamSqlReinterpretExpressionTest.java |  130 --
 .../operator/BeamSqlUdfExpressionTest.java         |   50 -
 .../BeamSqlArithmeticExpressionTest.java           |  332 ----
 .../operator/array/BeamSqlArrayExpressionTest.java |   80 -
 .../array/BeamSqlArrayItemExpressionTest.java      |   98 -
 .../BeamSqlCardinalityExpressionTest.java          |   94 -
 .../BeamSqlSingleElementExpressionTest.java        |   94 -
 .../date/BeamSqlCurrentDateExpressionTest.java     |   36 -
 .../date/BeamSqlCurrentTimeExpressionTest.java     |   40 -
 .../BeamSqlCurrentTimestampExpressionTest.java     |   40 -
 .../date/BeamSqlDateExpressionTestBase.java        |   36 -
 .../date/BeamSqlDatetimeMinusExpressionTest.java   |  150 --
 ...BeamSqlDatetimeMinusIntervalExpressionTest.java |  142 --
 .../date/BeamSqlDatetimePlusExpressionTest.java    |  186 --
 .../BeamSqlIntervalMultiplyExpressionTest.java     |  110 --
 ...eamSqlTimestampMinusIntervalExpressionTest.java |  170 --
 ...amSqlTimestampMinusTimestampExpressionTest.java |  210 --
 .../operator/date/TimeUnitUtilsTest.java           |   59 -
 .../operator/logical/BeamSqlNotExpressionTest.java |   55 -
 .../math/BeamSqlMathBinaryExpressionTest.java      |  289 ---
 .../math/BeamSqlMathUnaryExpressionTest.java       |  446 -----
 .../DatetimeReinterpretConversionsTest.java        |   68 -
 .../IntegerReinterpretConversionsTest.java         |   76 -
 .../reinterpret/ReinterpretConversionTest.java     |  106 -
 .../operator/reinterpret/ReinterpreterTest.java    |  180 --
 .../row/BeamSqlFieldAccessExpressionTest.java      |   91 -
 .../sql/impl/rel/BeamEnumerableConverterTest.java  |    5 +
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java     |   25 +
 .../sql/impl/utils/SqlTypeUtilsTest.java           |   76 -
 .../BeamSqlComparisonOperatorsIntegrationTest.java |   44 +-
 .../BeamSqlDateFunctionsIntegrationTest.java       |   32 +-
 .../beam/sdk/fn/channel/ManagedChannelFactory.java |   18 +-
 .../beam/sdk/fn/channel/SocketAddressFactory.java  |    2 +-
 .../data/BeamFnDataBufferingOutboundObserver.java  |    4 +-
 .../sdk/fn/data/BeamFnDataGrpcMultiplexer.java     |    4 +-
 .../beam/sdk/fn/data/RemoteGrpcPortRead.java       |    2 +-
 .../beam/sdk/fn/data/RemoteGrpcPortWrite.java      |    2 +-
 .../sdk/fn/splittabledofn/RestrictionTrackers.java |   90 +-
 .../sdk/fn/stream/BufferingStreamObserver.java     |    4 +-
 .../org/apache/beam/sdk/fn/stream/DataStreams.java |    2 +-
 .../beam/sdk/fn/stream/DirectStreamObserver.java   |    4 +-
 .../stream/ForwardingClientResponseObserver.java   |    6 +-
 .../sdk/fn/stream/OutboundObserverFactory.java     |    4 +-
 .../sdk/fn/stream/SynchronizedStreamObserver.java  |    2 +-
 .../fn/test/InProcessManagedChannelFactory.java    |    4 +-
 .../org/apache/beam/sdk/fn/test/TestStreams.java   |    4 +-
 .../sdk/fn/windowing/EncodedBoundedWindow.java     |    2 +-
 .../sdk/fn/channel/ManagedChannelFactoryTest.java  |    6 +-
 .../sdk/fn/channel/SocketAddressFactoryTest.java   |    2 +-
 .../BeamFnDataBufferingOutboundObserverTest.java   |    2 +-
 .../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java |    2 +-
 .../beam/sdk/fn/data/RemoteGrpcPortReadTest.java   |    2 +-
 .../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java  |    2 +-
 .../fn/splittabledofn/RestrictionTrackersTest.java |   72 +
 .../apache/beam/sdk/fn/stream/DataStreamsTest.java |    2 +-
 .../ForwardingClientResponseObserverTest.java      |    6 +-
 .../sdk/fn/stream/OutboundObserverFactoryTest.java |    4 +-
 .../sdk/fn/windowing/EncodedBoundedWindowTest.java |    2 +-
 sdks/java/harness/build.gradle                     |    2 +-
 .../beam/fn/harness/BoundedSourceRunner.java       |    2 +-
 .../java/org/apache/beam/fn/harness/FnHarness.java |    2 +-
 .../beam/fn/harness/PrecombineGroupingTable.java   |    2 +-
 .../harness/SplittableProcessElementsRunner.java   |    8 +-
 .../harness/control/AddHarnessIdInterceptor.java   |    8 +-
 .../fn/harness/control/BeamFnControlClient.java    |    4 +-
 .../fn/harness/control/ProcessBundleHandler.java   |   58 +-
 .../beam/fn/harness/control/RegisterHandler.java   |    2 +-
 .../beam/fn/harness/data/BeamFnDataGrpcClient.java |    2 +-
 .../fn/harness/data/QueueingBeamFnDataClient.java  |  182 ++
 .../fn/harness/logging/BeamFnLoggingClient.java    |   12 +-
 .../apache/beam/fn/harness/state/BagUserState.java |    2 +-
 .../harness/state/BeamFnStateGrpcClientCache.java  |    4 +-
 .../beam/fn/harness/state/FnApiStateAccessor.java  |    2 +-
 .../beam/fn/harness/state/MultimapSideInput.java   |    2 +-
 .../fn/harness/state/StateFetchingIterators.java   |    2 +-
 .../stream/HarnessStreamObserverFactories.java     |    2 +-
 .../beam/fn/harness/BoundedSourceRunnerTest.java   |    2 +-
 .../beam/fn/harness/FnApiDoFnRunnerTest.java       |  138 +-
 .../org/apache/beam/fn/harness/FnHarnessTest.java  |    6 +-
 .../harness/control/BeamFnControlClientTest.java   |    8 +-
 .../harness/control/ProcessBundleHandlerTest.java  |    2 +-
 .../fn/harness/data/BeamFnDataGrpcClientTest.java  |   14 +-
 .../data/BeamFnDataInboundObserverTest.java        |    2 +-
 ...Test.java => QueueingBeamFnDataClientTest.java} |  229 ++-
 .../harness/logging/BeamFnLoggingClientTest.java   |   16 +-
 .../beam/fn/harness/state/BagUserStateTest.java    |    2 +-
 .../state/BeamFnStateGrpcClientCacheTest.java      |   16 +-
 .../fn/harness/state/FakeBeamFnStateClient.java    |    2 +-
 .../fn/harness/state/MultimapSideInputTest.java    |    2 +-
 .../harness/state/StateFetchingIteratorsTest.java  |    2 +-
 .../stream/HarnessStreamObserverFactoriesTest.java |    4 +-
 .../beam/sdk/io/aws/options/AwsModuleTest.java     |    4 +-
 .../beam/sdk/io/aws/s3/S3ResourceIdTest.java       |    2 +-
 .../sdk/io/aws/s3/S3WritableByteChannelTest.java   |    2 +-
 .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java  |    3 +-
 .../apache/beam/sdk/io/cassandra/CassandraIO.java  |  161 +-
 .../beam/sdk/io/cassandra/CassandraService.java    |   16 +-
 .../sdk/io/cassandra/CassandraServiceImpl.java     |   76 +-
 .../beam/sdk/io/cassandra/CassandraIOTest.java     |   66 +-
 sdks/java/io/common/build.gradle                   |    2 +-
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |    4 +-
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |    4 +-
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |    4 +-
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java |   74 +-
 sdks/java/io/file-based-io-tests/build.gradle      |    3 +-
 .../io/common/FileBasedIOTestPipelineOptions.java  |   19 +
 .../java/org/apache/beam/sdk/io/text/TextIOIT.java |   44 +-
 .../apache/beam/sdk/io/gcp/bigquery/AvroUtils.java |    2 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java  |   10 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java       |    4 +-
 .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java  |   35 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryUtils.java    |    2 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java       |    2 +-
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java    |  136 +-
 .../beam/sdk/io/gcp/testing/BigqueryClient.java    |    8 +-
 .../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java |    2 +-
 .../io/gcp/bigquery/BigQueryServicesImplTest.java  |   88 +-
 .../sdk/io/gcp/bigquery/BigQueryToTableIT.java     |    6 +-
 .../beam/sdk/io/gcp/datastore/V1TestUtil.java      |    2 +-
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java       |  193 ++
 .../beam/sdk/io/gcp/spanner/OrderedCodeTest.java   |    2 +-
 .../build.gradle                                   |   55 +-
 .../io/hadoop/format/ExternalSynchronization.java  |   62 +
 .../sdk/io/hadoop/format/HDFSSynchronization.java  |  186 ++
 .../beam/sdk/io/hadoop/format/HadoopFormatIO.java  | 1987 +++++++++++++++++++
 .../beam/sdk/io/hadoop/format/HadoopFormats.java   |  243 +++
 .../sdk/io/hadoop/format/IterableCombinerFn.java   |  140 ++
 .../beam/sdk/io/hadoop/format}/package-info.java   |   10 +-
 .../format/ConfigurableEmployeeInputFormat.java    |  126 ++
 .../apache/beam/sdk/io/hadoop/format/Employee.java |   87 +
 .../sdk/io/hadoop/format}/EmployeeInputFormat.java |   10 +-
 .../sdk/io/hadoop/format/EmployeeOutputFormat.java |   73 +
 .../io/hadoop/format/HDFSSynchronizationTest.java  |  173 ++
 .../hadoop/format/HadoopFormatIOCassandraIT.java   |  197 ++
 .../hadoop/format/HadoopFormatIOCassandraTest.java |  235 +++
 .../io/hadoop/format/HadoopFormatIOElasticIT.java  |  220 +++
 .../hadoop/format/HadoopFormatIOElasticTest.java   |  277 +++
 .../sdk/io/hadoop/format/HadoopFormatIOIT.java     |  189 ++
 .../io/hadoop/format/HadoopFormatIOReadTest.java}  |  151 +-
 .../format/HadoopFormatIOSequenceFileTest.java     |  372 ++++
 .../hadoop/format/HadoopFormatIOTestOptions.java   |   76 +
 .../io/hadoop/format/HadoopFormatIOWriteTest.java  |  314 +++
 .../sdk/io/hadoop/format/IterableCombinerTest.java |   98 +
 .../format}/ReuseObjectsEmployeeInputFormat.java   |   10 +-
 .../sdk/io/hadoop/format}/TestEmployeeDataSet.java |    4 +-
 .../sdk/io/hadoop/format}/TestRowDBWritable.java   |   13 +-
 .../src/test/resources/cassandra.yaml              |    0
 sdks/java/io/hadoop-input-format/build.gradle      |    1 +
 .../io/hadoop/inputformat/HadoopInputFormatIO.java |  871 +--------
 .../io/hadoop/inputformat/EmployeeInputFormat.java |    6 +-
 .../inputformat/HadoopInputFormatIOTest.java       |  462 -----
 .../ReuseObjectsEmployeeInputFormat.java           |    4 +-
 .../io/hadoop/inputformat/TestEmployeeDataSet.java |    2 +-
 .../io/hadoop/inputformat/TestRowDBWritable.java   |    9 +-
 .../beam/sdk/io/hbase/HBaseReadSplittableDoFn.java |    3 +-
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   |    3 +-
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java     |   29 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |   13 +
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  |    4 +-
 .../beam/sdk/io/kinesis/ShardReadersPool.java      |   25 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  |    2 +-
 sdks/java/io/kudu/build.gradle                     |    2 +-
 sdks/java/io/mongodb/build.gradle                  |    6 +-
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java       |    4 +-
 .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java  |  186 +-
 .../org/apache/beam/sdk/io/mongodb/SSLUtils.java   |   75 +
 .../beam/sdk/io/mongodb/MongoDBGridFSIOTest.java   |    2 +-
 .../apache/beam/sdk/io/mongodb/MongoDbIOTest.java  |   93 +
 sdks/java/io/rabbitmq/build.gradle                 |    2 +-
 sdks/java/io/synthetic/build.gradle                |    2 +-
 .../beam/sdk/io/synthetic/SyntheticOptions.java    |    8 +
 sdks/java/io/tika/build.gradle                     |    2 +-
 .../java/org/apache/beam/sdk/io/xml/XmlIO.java     |    4 +-
 sdks/java/javadoc/build.gradle                     |   69 +-
 sdks/java/maven-archetypes/examples/build.gradle   |    2 +-
 .../src/main/resources/archetype-resources/pom.xml |   28 +
 sdks/java/maven-archetypes/starter/build.gradle    |    2 +-
 sdks/java/testing/load-tests/build.gradle          |    3 +-
 .../beam/sdk/loadtests/CoGroupByKeyLoadTest.java   |   22 +-
 .../apache/beam/sdk/loadtests/CombineLoadTest.java |   22 +-
 .../beam/sdk/loadtests/ConsoleResultPublisher.java |   15 +-
 .../beam/sdk/loadtests/GroupByKeyLoadTest.java     |   13 +-
 .../org/apache/beam/sdk/loadtests/LoadTest.java    |   64 +-
 .../apache/beam/sdk/loadtests/LoadTestOptions.java |   17 +
 .../apache/beam/sdk/loadtests/LoadTestResult.java  |   67 +
 .../apache/beam/sdk/loadtests/ParDoLoadTest.java   |    9 +-
 .../loadtests/SyntheticDataPubSubPublisher.java    |  107 +
 .../{MetricsPublisher.java => ByteMonitor.java}    |   30 +-
 .../{MetricsMonitor.java => TimeMonitor.java}      |   20 +-
 sdks/java/testing/nexmark/build.gradle             |    2 +-
 .../java/org/apache/beam/sdk/nexmark/Main.java     |   39 +-
 .../beam/sdk/nexmark/NexmarkConfiguration.java     |    4 +
 .../apache/beam/sdk/nexmark/NexmarkLauncher.java   |   62 +-
 .../org/apache/beam/sdk/nexmark/NexmarkPerf.java   |   11 +-
 .../apache/beam/sdk/nexmark/NexmarkQueryName.java  |    3 +-
 .../org/apache/beam/sdk/nexmark/NexmarkUtils.java  |   58 +-
 .../sdk/nexmark/queries/SessionSideInputJoin.java  |  106 +
 .../nexmark/queries/SessionSideInputJoinModel.java |  151 ++
 .../sdk/nexmark/sources/generator/Generator.java   |    3 +-
 .../sources/generator/model/PersonGenerator.java   |    8 +-
 .../beam/sdk/nexmark/PerfsToBigQueryTest.java      |   50 +-
 .../nexmark/queries/SessionSideInputJoinTest.java  |  212 ++
 sdks/java/testing/test-utils/build.gradle          |    2 +-
 .../apache/beam/sdk/testutils/NamedTestResult.java |   76 +
 .../org/apache/beam/sdk/testutils/TestResult.java  |   16 +-
 .../beam/sdk/testutils/metrics/MetricsReader.java  |   97 +-
 .../apache/beam/sdk/testutils}/package-info.java   |    5 +-
 .../sdk/testutils/publishing/BigQueryClient.java   |   12 +
 .../publishing/BigQueryResultsPublisher.java       |   60 +
 .../sdk/testutils/fakes/FakeBigQueryClient.java    |    4 +-
 ...ient.java => FakeBigQueryResultsPublisher.java} |   44 +-
 .../sdk/testutils/metrics/MetricsReaderTest.java   |   52 +-
 .../publishing/BigQueryResultsPublisherTest.java   |  102 +
 sdks/python/apache_beam/coders/coder_impl.pxd      |   65 +-
 sdks/python/apache_beam/coders/coder_impl.py       |  169 +-
 sdks/python/apache_beam/coders/coders.py           |  147 +-
 .../apache_beam/coders/coders_test_common.py       |   52 +-
 .../cookbook/bigquery_tornadoes_it_test.py         |    2 +-
 .../apache_beam/examples/snippets/snippets.py      |    5 +-
 sdks/python/apache_beam/internal/http_client.py    |   70 +
 .../apache_beam/internal/http_client_test.py       |  109 ++
 sdks/python/apache_beam/io/__init__.py             |    1 +
 sdks/python/apache_beam/io/filebasedsource_test.py |   20 +-
 sdks/python/apache_beam/io/filesystem.py           |    4 +-
 sdks/python/apache_beam/io/filesystem_test.py      |   23 +-
 sdks/python/apache_beam/io/filesystemio.py         |    4 +-
 sdks/python/apache_beam/io/filesystemio_test.py    |   33 +-
 sdks/python/apache_beam/io/gcp/bigquery.py         |   18 +-
 sdks/python/apache_beam/io/gcp/bigquery_test.py    |   21 +-
 sdks/python/apache_beam/io/gcp/gcsio.py            |  174 +-
 .../apache_beam/io/gcp/gcsio_integration_test.py   |  183 ++
 sdks/python/apache_beam/io/gcp/gcsio_test.py       |   48 +-
 sdks/python/apache_beam/io/gcp/pubsub_test.py      |  106 +-
 sdks/python/apache_beam/io/parquetio.py            |  472 +++++
 sdks/python/apache_beam/io/parquetio_it_test.py    |  176 ++
 sdks/python/apache_beam/io/parquetio_test.py       |  463 +++++
 sdks/python/apache_beam/io/range_trackers.py       |   28 +-
 sdks/python/apache_beam/io/range_trackers_test.py  |  133 +-
 .../apache_beam/io/source_test_utils_test.py       |   29 +-
 sdks/python/apache_beam/io/sources_test.py         |   18 +-
 sdks/python/apache_beam/io/textio.py               |   27 +-
 sdks/python/apache_beam/io/textio_test.py          |   26 +
 sdks/python/apache_beam/io/tfrecordio_test.py      |    9 +-
 .../python/apache_beam/options/pipeline_options.py |    8 +
 .../apache_beam/options/pipeline_options_test.py   |   17 +
 sdks/python/apache_beam/pipeline.py                |   13 +-
 sdks/python/apache_beam/portability/python_urns.py |   16 +
 .../runners/dataflow/dataflow_runner.py            |  106 +-
 .../runners/dataflow/internal/apiclient.py         |   51 +-
 .../apache_beam/runners/dataflow/internal/names.py |    4 +-
 .../runners/dataflow/test_dataflow_runner.py       |   13 +-
 .../apache_beam/runners/direct/direct_runner.py    |   14 +-
 .../runners/direct/test_direct_runner.py           |   10 +-
 .../runners/interactive/interactive_runner.py      |   12 +-
 .../python/apache_beam/runners/pipeline_context.py |    7 +-
 .../runners/portability/flink_runner_test.py       |   93 +-
 .../runners/portability/fn_api_runner.py           | 1277 ++++--------
 .../runners/portability/fn_api_runner_test.py      |   69 +-
 .../portability/fn_api_runner_transforms.py        |  948 +++++++++
 .../apache_beam/runners/portability/job_server.py  |   27 +-
 .../runners/portability/local_job_service.py       |   84 +-
 .../runners/portability/local_job_service_main.py  |    4 +-
 .../runners/portability/portable_runner.py         |  153 +-
 .../runners/portability/portable_runner_test.py    |   45 +-
 .../apache_beam/runners/portability/stager.py      |    3 +-
 sdks/python/apache_beam/runners/runner.py          |   14 +-
 .../apache_beam/runners/worker/bundle_processor.py |   90 +-
 .../apache_beam/runners/worker/data_plane.py       |   14 +-
 .../apache_beam/runners/worker/operations.py       |    2 +
 .../apache_beam/runners/worker/sdk_worker.py       |   77 +-
 .../runners/worker/worker_id_interceptor.py        |    5 +-
 .../testing/load_tests/co_group_by_key_test.py     |   75 +-
 .../apache_beam/testing/load_tests/combine_test.py |   69 +-
 .../testing/load_tests/group_by_key_test.py        |   76 +-
 .../testing/load_tests/load_test_metrics_utils.py  |  165 +-
 .../apache_beam/testing/load_tests/pardo_test.py   |  132 +-
 .../testing/load_tests/sideinput_test.py           |  203 ++
 .../apache_beam/testing/synthetic_pipeline.py      |    2 +-
 .../apache_beam/testing/synthetic_pipeline_test.py |    2 +-
 sdks/python/apache_beam/testing/test_utils.py      |    7 +-
 .../apache_beam/tools/coders_microbenchmark.py     |   50 +-
 sdks/python/apache_beam/transforms/core.py         |    6 +-
 .../apache_beam/transforms/ptransform_test.py      |  114 +-
 .../apache_beam/transforms/userstate_test.py       |   32 +-
 sdks/python/apache_beam/transforms/util.py         |   18 +-
 sdks/python/apache_beam/transforms/util_test.py    |    8 +
 sdks/python/apache_beam/transforms/window.py       |   26 +-
 sdks/python/apache_beam/utils/proto_utils.py       |    8 +-
 sdks/python/apache_beam/utils/windowed_value.pxd   |   19 +
 sdks/python/apache_beam/utils/windowed_value.py    |   61 +-
 sdks/python/build.gradle                           |  161 +-
 sdks/python/container/base_image_requirements.txt  |    2 +-
 sdks/python/container/build.gradle                 |    4 +-
 .../python/precommit/dataflow}/build.gradle        |   27 +-
 sdks/python/scripts/generate_pydoc.sh              |    1 +
 sdks/python/scripts/run_integration_test.sh        |   20 +-
 sdks/python/setup.py                               |   24 +-
 sdks/python/tox.ini                                |    4 +-
 settings.gradle                                    |   20 +
 vendor/grpc-1_13_1/build.gradle                    |    4 +-
 vendor/sdks-java-extensions-protobuf/build.gradle  |   13 +-
 website/Gemfile.lock                               |   16 +-
 website/Rakefile                                   |   13 +-
 website/_config.yml                                |    2 +-
 website/src/.htaccess                              |    2 +-
 website/src/_data/authors.yml                      |    3 +
 website/src/_includes/section-menu/community.html  |    1 +
 website/src/_posts/2017-01-09-added-apex-runner.md |    4 +-
 website/src/_posts/2018-12-13-beam-2.9.0.md        |   62 +
 website/src/community/contact-us.md                |    2 +-
 website/src/community/in-person.md                 |   47 +
 website/src/contribute/committer-guide.md          |   37 +-
 website/src/contribute/index.md                    |   25 +-
 website/src/contribute/postcommits-guides.md       |    2 +-
 .../src/contribute/postcommits-policies-details.md |   18 +
 website/src/contribute/ptransform-style-guide.md   |    4 +-
 website/src/contribute/release-guide.md            |    2 +-
 .../documentation/io/built-in-google-bigquery.md   |    2 +-
 website/src/documentation/runners/apex.md          |    2 +-
 website/src/documentation/runners/flink.md         |    2 +-
 website/src/documentation/sdks/euphoria.md         |    4 +-
 .../src/documentation/sdks/java-dependencies.md    |  497 +++--
 website/src/documentation/sdks/nexmark.md          |    7 +-
 .../src/documentation/sdks/python-dependencies.md  |   64 +-
 website/src/get-started/downloads.md               |    7 +
 997 files changed, 28980 insertions(+), 21156 deletions(-)
 rename .test-infra/jenkins/{job_PostCommit_Go_GradleBuild.groovy => job_PostCommit_Go.groovy} (91%)
 copy .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy => job_PostCommit_Java.groovy} (92%)
 rename .test-infra/jenkins/{job_PostCommit_Java_PortabilityApi_GradleBuild.groovy => job_PostCommit_Java_PortabilityApi.groovy} (94%)
 copy .test-infra/jenkins/{job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy => job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy} (83%)
 rename .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy => job_PostCommit_SQL.groovy} (78%)
 rename sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/package-info.java => .test-infra/jenkins/job_PreCommit_Portable_Python.groovy (71%)
 rename .test-infra/jenkins/{job_Release_Gradle_NightlySnapshot.groovy => job_Release_NightlySnapshot.groovy} (96%)
 copy runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{SerializablePipelineOptions.java => PipelineOptionsSerializationUtils.java} (58%)
 create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExecutableStageTranslationTest.java
 create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
 rename runners/{samza/src/main/java/org/apache/beam/runners/samza/util => core-java/src/main/java/org/apache/beam/runners/core/serialization}/Base64Serializer.java (94%)
 rename {sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date => runners/core-java/src/main/java/org/apache/beam/runners/core/serialization}/package-info.java (89%)
 create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilderTest.java
 copy {model/pipeline => runners/flink/1.6}/build.gradle (59%)
 copy {sdks/java/build-tools => runners/flink/1.6/job-server-container}/build.gradle (79%)
 copy {model/pipeline => runners/flink/1.6/job-server}/build.gradle (65%)
 copy runners/flink/{build.gradle => flink_runner.gradle} (89%)
 copy runners/flink/job-server-container/{build.gradle => flink_job_server_container.gradle} (61%)
 copy runners/flink/job-server/{build.gradle => flink_job_server.gradle} (50%)
 rename runners/flink/src/main/java/org/apache/beam/runners/flink/{PipelineTranslationOptimizer.java => PipelineTranslationModeOptimizer.java} (79%)
 create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FileReporter.java
 create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/Metrics.java
 create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/utils/NoopLock.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTransformOverridesTest.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/PipelineTranslationModeOptimizerTest.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByWithNullValuesTest.java
 create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/FlinkPipelineTranslatorUtilsTest.java
 rename runners/flink/src/test/java/org/apache/beam/runners/flink/{streaming => translation/wrappers/streaming/io}/TestCountingSource.java (94%)
 rename runners/flink/src/test/java/org/apache/beam/runners/flink/{streaming => translation/wrappers/streaming/io}/UnboundedSourceWrapperTest.java (82%)
 rename runners/google-cloud-dataflow-java/{worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/IdGeneratorTest.java => src/test/java/org/apache/beam/runners/dataflow/util/GCSUploadMain.java} (55%)
 delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/ServerFactory.java
 delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactory.java
 create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ProcessRemoteBundleOperation.java
 copy runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/{RegisterNodeFunction.java => CreateExecutableStageNodeFunction.java} (56%)
 delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/ServerFactoryTest.java
 delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactoryTest.java
 copy runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/{ProcessEnvironmentFactory.java => ExternalEnvironmentFactory.java} (59%)
 create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/StaticRemoteEnvironment.java
 create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/StaticRemoteEnvironmentFactory.java
 create mode 100644 runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
 rename sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlOperator.java => runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnInvokerRegistrar.java (57%)
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/GroupWithoutRepartition.java
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueGetterFactory.java => runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/UpdatingCombineFn.java (57%)
 rename {sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic => runners/samza/src/main/java/org/apache/beam/runners/samza/transforms}/package-info.java (87%)
 rename runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/{ => mocks}/DatasetSourceMockBatch.java (72%)
 rename runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/{ => mocks}/ReadSourceTranslatorMockBatch.java (81%)
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn_arity.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn_arity.tmpl
 copy sdks/go/pkg/beam/core/util/ioutilx/{read.go => write.go} (67%)
 create mode 100644 sdks/go/pkg/beam/core/util/reflectx/structs.go
 copy sdks/go/pkg/beam/{core/util/ioutilx/read.go => util/gcsx/gcs_test.go} (63%)
 create mode 100644 sdks/java/bom/build.gradle
 create mode 100644 sdks/java/bom/pom.xml.template
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsOptions.java
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java
 rename sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/package-info.java => core/src/main/java/org/apache/beam/sdk/schemas/Factory.java} (76%)
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
 rename sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/package-info.java => core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformationFactory.java} (71%)
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java
 rename sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/package-info.java => core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeCreator.java} (73%)
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SetterBasedCreatorFactory.java
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java (76%)
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
 rename sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/{PojoValueGetterFactory.java => FieldValueTypeSupplier.java} (64%)
 delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanGetterFactory.java
 delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanSetterFactory.java
 delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/PojoValueSetterFactory.java
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/core/src/main/java/org/apache/beam/sdk/testing/DataflowPortabilityExecutableStageUnsupported.java (69%)
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputs.java (77%)
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/{SimpleFunction.java => InferableFunction.java} (64%)
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/{SerializableFunction.java => ProcessFunction.java} (51%)
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Restrictions.java (65%)
 create mode 100644 sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc
 create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
 create mode 100644 sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpCallCustomError.java (70%)
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpCallMatcher.java (67%)
 rename sdks/java/extensions/{sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionExecutor.java => google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpRequestWrapper.java} (54%)
 rename runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/IdGenerator.java => sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpResponseWrapper.java (55%)
 create mode 100644 sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/util/CustomHttpErrorsTest.java
 copy sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{UdfImpl.java => ScalarFunctionImpl.java} (84%)
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionEnvironment.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionEnvironments.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlBinaryOperator.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCaseExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCastExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDefaultExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDotExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlInputRefExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlLocalRefExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlOperatorExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitive.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUdfExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUnaryOperator.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/DateOperators.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/StringOperators.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlArithmeticExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlDivideExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlPlusExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayItemExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlCardinalityExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlSingleElementExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlCompareExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlEqualsExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlGreaterThanExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlIsNullExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLessThanExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLessThanOrEqualsExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLikeExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlNotEqualsExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlNotLikeExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentDateExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimeExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimestampExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusIntervalExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimePlusExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlIntervalMultiplyExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusIntervalExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusTimestampExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/TimeUnitUtils.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlAndExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlNotExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlOrExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/BeamSqlMapExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/BeamSqlMapItemExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAbsExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAcosExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAsinExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAtan2Expression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAtanExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCeilExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCosExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlExpExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlFloorExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlLnExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlLogExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathBinaryExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathUnaryExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlPiExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlPowerExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRadiansExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRandExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRandIntegerExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRoundExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlSignExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlSinExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlTanExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlTruncateExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/package-info.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/package-info.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/BeamSqlReinterpretExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/DatetimeReinterpretConversions.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/IntegerReinterpretConversions.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpretConversion.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/Reinterpreter.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/package-info.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/BeamSqlFieldAccessExpression.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/package-info.java
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/package-info.java
 rename sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{interpreter/operator/logical/BeamSqlLogicalExpression.java => planner/BeamJavaTypeFactory.java} (50%)
 delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SqlTypeUtils.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTestBase.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamNullExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlAndOrExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCaseExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCastExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCompareExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDotExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlInputRefExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitiveTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlReinterpretExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUdfExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayItemExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlCardinalityExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlSingleElementExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentDateExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDateExpressionTestBase.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusIntervalExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimePlusExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlIntervalMultiplyExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusIntervalExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusTimestampExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/TimeUnitUtilsTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlNotExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathBinaryExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathUnaryExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/DatetimeReinterpretConversionsTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/IntegerReinterpretConversionsTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpretConversionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpreterTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/BeamSqlFieldAccessExpressionTest.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/SqlTypeUtilsTest.java
 create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
 copy sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/{BeamFnDataGrpcClientTest.java => QueueingBeamFnDataClientTest.java} (63%)
 copy sdks/java/io/{hadoop-input-format => hadoop-format}/build.gradle (66%)
 create mode 100644 sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/ExternalSynchronization.java
 create mode 100644 sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java
 create mode 100644 sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
 create mode 100644 sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java
 create mode 100644 sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/IterableCombinerFn.java
 copy sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical => io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format}/package-info.java (76%)
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ConfigurableEmployeeInputFormat.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/Employee.java
 copy sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat => hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/EmployeeInputFormat.java (93%)
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeOutputFormat.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronizationTest.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOCassandraIT.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOCassandraTest.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticIT.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java
 copy sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java => hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java} (85%)
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOSequenceFileTest.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOTestOptions.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOWriteTest.java
 create mode 100644 sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/IterableCombinerTest.java
 copy sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat => hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/ReuseObjectsEmployeeInputFormat.java (93%)
 copy sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat => hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/TestEmployeeDataSet.java (97%)
 copy sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat => hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/TestRowDBWritable.java (88%)
 copy sdks/java/io/{hadoop-input-format => hadoop-format}/src/test/resources/cassandra.yaml (100%)
 create mode 100644 sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/SSLUtils.java
 copy runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ConsoleResultPublisher.java (69%)
 create mode 100644 sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java
 create mode 100644 sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/SyntheticDataPubSubPublisher.java
 rename sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/metrics/{MetricsPublisher.java => ByteMonitor.java} (51%)
 rename sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/metrics/{MetricsMonitor.java => TimeMonitor.java} (69%)
 create mode 100644 sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java
 create mode 100644 sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java
 create mode 100644 sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java
 create mode 100644 sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
 rename runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java => sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/TestResult.java (70%)
 rename sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical => testing/test-utils/src/main/java/org/apache/beam/sdk/testutils}/package-info.java (88%)
 create mode 100644 sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisher.java
 copy sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/fakes/{FakeBigQueryClient.java => FakeBigQueryResultsPublisher.java} (52%)
 create mode 100644 sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java
 create mode 100644 sdks/python/apache_beam/internal/http_client.py
 create mode 100644 sdks/python/apache_beam/internal/http_client_test.py
 create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
 create mode 100644 sdks/python/apache_beam/io/parquetio.py
 create mode 100644 sdks/python/apache_beam/io/parquetio_it_test.py
 create mode 100644 sdks/python/apache_beam/io/parquetio_test.py
 create mode 100644 sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
 create mode 100644 sdks/python/apache_beam/testing/load_tests/sideinput_test.py
 copy {model/job-management => sdks/python/precommit/dataflow}/build.gradle (50%)
 create mode 100644 website/src/_posts/2018-12-13-beam-2.9.0.md
 create mode 100644 website/src/community/in-person.md


Mime
View raw message