beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bhule...@apache.org
Subject [beam] branch website-revamp updated (8b22fb4 -> 7d0d220)
Date Wed, 13 Jan 2021 01:56:14 GMT
This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.


    from 8b22fb4  Merge remote-tracking branch 'origin/master' into website-revamp
     add ce69ec6  [BEAM-11312] Minor improvement in retrieving build log (#13501)
     add 196fd9d  [BEAM-11207] Return valid error on job failures (#13484)
     add 0498a17  [BEAM-11413] Fix default job server for Spark tests.
     add 7b39702  Merge pull request #13498 from ibzib/BEAM-11413
     add 80eff6c  [BEAM-11417] Use Cache with time eviction policy for commitCallbacks
     add a9a1c8e  Merge pull request #13507 from [BEAM-11417] Use Cache with time eviction policy for commitCallbacks
     add bbaaeb0  Decode data channel elements after, rather than before, placing them in the queue. (#13387)
     add 5caeb5d  Merge pull request #13395 from [BEAM-11426] Add FHIR Search to io/gcp/healthcare/FhirIO
     add 05c8471  Add GroupIntoBatches to runner API; add Dataflow override in Python SDK
     add 2c96aeb  Merge pull request #13405 from [BEAM-10703, BEAM-10475] Add GroupIntoBatches to runner API; add Dataflow override in Python SDK
     add 62f385c  [BEAM-11419] Add Flink 1.12 build target.
     add a35513d  Merge pull request #13504: [BEAM-11419] Add Flink 1.12 build target
     add 15f9270  [BEAM-11075] Fix go sdk gbk load test (#13494)
     add e1cff1a  [BEAM-11400] Change StreamingDataflowWorker to ignore commits that don't match.
     add dce9158  Merge pull request #13486: [BEAM-11400] Change StreamingDataflowWorker to ignore commits that do…
     add b39602a  [BEAM-11384] Change StreamingDataflowWorker work id for logs to sharding key.
     add 5f50f17  Merge pull request #13459: [BEAM-11384] Change StreamingDataflowWorker from encoding key
     add adf5515  Better logging for Python Docker environment (#13508)
     add a2815b5  [BEAM-11415] Setup and bugfixes for Go ValidatesRunner framework.
     add 310a710  Merge pull request #13502: [BEAM-11415] Setup and bugfixes for Go ValidatesRunner framework.
     add b942279  [BEAM-10114] Bump Pub/Sub lite version and clean up settings classes.
     add c1d9794  [BEAM-10114] Fix lint errors
     add bc0bac7  Merge pull request #13509 from [BEAM-10114] Bump Pub/Sub lite version and clean up settings classes.
     add e8a9c7b  [BEAM-11415] In-progress Go ValidatesRunner framework.
     add d009fe9  Merge pull request #13503: [BEAM-11415] In-progress Go ValidatesRunner framework.
     add cd2c312  [BEAM-11435] Reuse already set timers.
     add 744637d  Merge pull request #13522: [BEAM-11435] Reuse already set timers.
     add 190261a  Log exception caught during UnboundedSource#split
     add 76723d8  Merge pull request #13520: Log exception caught during UnboundedSource#split
     add 01a63c7  [BEAM-10556][BEAM-10402] Cleanup DoFnOperatorTest warnings.
     add 33b3ac4  Merge pull request #13521: [BEAM-10556][BEAM-10402] Cleanup DoFnOperatorTest warnings
     add 2356367  [BEAM-11425] Metrics extraction via Monitoring API (#13505)
     add aa1498c  [BEAM-11374] Add key regex filter to Bigtable for BeamSQL
     add d4e9171  Merge pull request #13495 from piotr-szuberski/bigtable-filters
     add 5e1c544  Add SDF capability_matrix.
     add 1481977  Merge pull request #13368 from [BEAM-10480] Add SDF capability_matrix.
     add 379d796  Add a small announcement for Splittable DoFn.
     add c4af7f9  Merge pull request #13456 from [BEAM-10480] Add a small announcement for Splittable DoFn.
     add 44fc9fe  [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings.
     add 0424485  Merge pull request #13525: [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings.
     add 0353554  [BEAM-11337] Make the ThriftCoder class visible from outside its package
     add 9317a6b  Merge pull request #13415: [BEAM-11337] Make the ThriftCoder class visible from outside its package
     add fd56cf1  [BEAM-7794] Enable DynamoDBIO tests (Use localstack version 0.11.4)
     add b850e76  Merge pull request #13510: [BEAM-7794] Enable DynamoDBIO tests (Use localstack version 0.11.4)
     add bf3c7eb  Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
     add 32598ff  Merge pull request #13233: [BEAM-11016] Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
     add 77683be  [BEAM-11439] Fix Snowflake streaming docs (#13528)
     add a679bab  [BEAM-9602] Add timer family support for python SDK
     add 8cf88c3  Merge pull request #13421 from [BEAM-9602] Add timer family support for python SDK
     add 566ee5e  [BEAM-10706] Fix duplicate key error in DynamoDBIO.Write
     add a1676e9  Merge pull request #12583: [BEAM-10706] Fix duplicate key error in DynamoDBIO.Write
     add 20bab56  Add myself (lostluck) to the authors.yml list (#13531)
     add 1c453dc  [BEAM-11208] Fix for QUOTA_EXCEEDED failures in BQ Storage streams sp… (#13378)
     add 2733e95  [BEAM-11434]Expose batch/admin clients in Spanner Accessor
     add 70ac878  Merge pull request #13519: [BEAM-11434]Expose batch/admin clients in Spanner Accessor
     add f36cd5b  Update Beam website to release 2.26.0. (#13481)
     add fdbf47d  Add a blogpost for ApacheBeam 2.26.0. (#13482)
     add dc8d82e  Fix typo in publish_docker_images.sh (#13532)
     add b9cc076  [BEAM-11338] Beam schema for thrift data (#13428)
     add ee911d2  Fix missing braket in the 2.25 signature. (#13534)
     add 302de87  Workaround for incremental in read_json. (#13489)
     add dc7ecb9  [BEAM-9196] Update testcontainers to 1.15.1
     add fc42781  Merge pull request #13530: [BEAM-9196] Update testcontainers to 1.15.1
     add 82bbdf7  [BEAM-10124] Add missing announcement from CHANGES.md (#13537)
     add 38d3c5d  [BEAM-11075] Enable Go load tests for Dataflow (#13529)
     add 755eab2  [BEAM-11338] Addendum: handle private fields in generated thrift code (#13539)
     add 7e85cc7  Add output type hints for GroupIntoBatches. (#13493)
     add 7bdc297  [BEAM-11444] Python IO MongoDB: mask password in display_data uri and spec. (#13541)
     add 18b0ee8  Change the output type of FHIR Search from String to JsonArray (#13516)
     add 8649972  [BEAM-11429] User-friendly names for packed combiners (#13535)
     add be85be2  [Beam-11438] start-build-env.sh improve for macOS (#13545)
     add 34ae21b  [BEAM-11065] Apache Beam pipeline example to ingest from Apache Kafka to Google Pub/Sub  (#13112)
     add 52a2270  [BEAM-11196] Fix `None` parent when fusing >2 stages (#13549)
     add f23ebad  [BEAM-10641] Add Combiner Packing to Dataflow (#13455)
     add 75e167e  [BEAM-11460] Implement reading unknown schema files for ParquetIO
     add 0686879  Update ParquertIO capability in CHANGES.md
     add de6d268  improve backward compatibility by creating separate `Parse<T>` and `ParseFiles<T>` implementation for supporting files with unknown schema.
     add 431831b  Fix review comments by @iemejia
     add 6faeda3  Make GenericRecordPassthroughFn singleton
     add afe11c6  fix spotless Apply
     add eb16936  [BEAM-11230] Fix a bug with decoding repeated fields (#13544)
     add 447d9ae  Enable more tests on Java + Python FnRunner
     add 531cded  Merge pull request #13547 from [BEAM-10450][BEAM-10449][BEAM-10453][BEAM-10995] Enable more Java tests on Python FnRunner
     add ba11ed7  Do not add unnecessary experiment use_multiple_sdk_containers. (#13475)
     add 7e09870  Skip dynamic timer test in portable spark test
     add 5878cc5  Merge pull request #13553 from Skip dynamic timer test in portable spark test
     add 3d62aab  Bump up python container versions
     add 3710357  Merge pull request #13552 from Bump up python container versions
     add 9f97585  [BEAM-11360] Updates Dataflow Python multi-language pipelines to use portable job submission by default (#13536)
     add 8dd9b38  [BEAM-11401][BEAM-11366] Change ReaderCache invalidations to be asynchronous and make the timeout configurable.
     add 0548643  Merge pull request #13485: [BEAM-11401][BEAM-11366] Change ReaderCache invalidations to be async…
     add 5184eac  Revert "Do not add unnecessary experiment use_multiple_sdk_containers." (#13555)
     add c60b70e  Adding limitations of SchemaUpdateOptions to BQIO javadoc.
     add 5d9f4de  Merge pull request #13551 from Adding limitations of SchemaUpdateOptions to BQIO javadoc.
     add 15ffb07  [BEAM-11472] Support SDF on non-splitting runners such as legacy Dataflow.
     add f137246  Merge pull request #13556 [BEAM-11472] Support SDF on non-splitting runners such as legacy Dataflow.
     add 31dcd53  [BEAM-10114] Convert PubsubLiteIO read to use SplittableDoFn.
     add ec3efe2  Fix test.
     add d3336fe  fix: Testing issues
     add 8892c1c  [BEAM-10114] Convert PubsubLiteIO to use an SDF for reads.
     add f5c6108  [BEAM-10114] Add byte and time limiting
     add 6d8c0a9  [BEAM-10114] Add byte and time limiting
     add 762b7ba  [BEAM-10114] Add byte and time limiting
     add 65b2c42  [BEAM-10114] Add byte and time limiting
     add 418b3ac  [BEAM-10114] Add byte and time limiting
     add 843fc4a  [BEAM-10114] Add byte and time limiting
     add fa3fc2d  fix: Linter issues
     add ea33f9e  fix: linters
     add 88a3021  fix: linters
     add b731da8  fix: Remove unnecessary shuffle.
     add a432e8f  fix: Address comments
     add f87c984  Merge pull request #13470 from [BEAM-10114] Convert PubsubLiteIO read to use SplittableDoFn.
     add dee8d49  [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner.
     add 56d4ac9  Merge pull request #13538: [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner.
     add ded0325  [BEAM-11075] Go SDK SideInput load tests (#13436)
     add f1c1514  [BEAM-11267] Remove unecessary reshuffle for stateful ParDo after keyed operation.
     add 76c6129  Merge pull request #13353: [BEAM-11267] Remove unnecessary reshuffle for stateful ParDo after key…
     add 12b6015  Update DoFn docstring (#13559)
     add cf20794  Fix concat source fraction_consumed() in degenerate case.
     add 8f79065  Merge pull request #13563 from robertwb/concat-isdone
     add b2f4784  [BEAM-11361] Dynamic splitting of dataframe csv reads. (#13488)
     add 97ad770  fix PerPartitionSubscriptionSdf methods with incorrect visibility
     add 92209d1  Merge pull request #13566 from fix PerPartitionSubscriptionSdf methods with incorrect visibility
     add 1ccdee8  [BEAM-11432] put latest tag when publishing SDK head containers to gcr (#13517)
     add 8de3288  [BEAM-10925] Add interface for SQL Java scalar UDF. (#13305)
     add 554d254  Track transform processing thread in Java SDK harness and set log entry transform id field.
     add a6ba301  Add javadoc
     add 23b2007  Add tracking to other ProcessElement methods
     add 945e794  Use LoadingCache instead of ConcurrentHashMap to limit the size for thread tracker
     add 857d41f  Address comment
     add c8d9d1d  Fix checkstyle
     add 6869dfa  Merge pull request #13533 from [BEAM-11474] Track transform processing thread in Java SDK harness and set log entry field
     add 3de4f88  [BEAM-10925] Add SQL AggregateFn interface.
     add a7457c0  Merge pull request #13306 from ibzib/aggregate-udf
     add 3cacd4a  [BEAM-11407] Add IT test to Bigtable for BeamSQL
     add 1a5505b  Merge pull request #13512 from piotr-szuberski/bigtable-it
     add 8fa3e87  Thread safe addition of new metrics
     add c7efdc1  Thread safe addition of new metrics
     add 5f253b6  Merge pull request #13526 from [BEAM-11459] Thread safe addition of new metrics
     add b8c001c  Moving to 2.28.0-SNAPSHOT on master branch.
     add 9c69229  [Beam-11002] Fixes BufferOverflowException in XMLReader (#13513)
     add db84863  Adding combiner packing optimization for Portable Runner
     add 2202581  fix formatting
     add fc84d02  Merge pull request #13573 from pabloem/cl151
     add 4f6f923  [BEAM-10658] Update website to recommend the right version
     add 0757359  Merge pull request #13576: [BEAM-10658] Update website to recommend the right version
     add f8d326e  Rewrite container docs for custom containers (#13420)
     add 77ff32c  Cleanup directory with SVN repo for PMC members
     add 1542971  Merge pull request #13580 from Cleanup directory with SVN repo for PMC members
     add 3663e03  [BEAM-11458] Upgrade SamzRunner to Samza 1.5 (#13550)
     add 9732fa3  Add DataFrame Preview announcment blog post (#13561)
     add 019db83  Dataframe docs (#13560)
     add 7d88858  [BEAM-11300] Improve Nexmark performance:
     add 80248d0  Merge pull request #13318: [BEAM-11300] Improve Nexmark performance:
     add 8de6faa  [BEAM-11481] emit output watermark on watermark hold change
     add bea6f88  Merge pull request #13571: [BEAM-11481] emit output watermark on watermark hold change
     add 287f77a  [BEAM-11329]:HDFS not deduplicating identical configuration paths (#13514)
     add 4b1c4aa  [BEAM-10925] SQL AggregateFn: Separate mutable accumulator from immutables.
     add 199fea8  Merge pull request #13579 from ibzib/accumulators
     add c4b1359  [BEAM-10925] Create ZetaSQL-specific subclass of ScalarFunctionImpl that knows which function group it belongs to.
     add 542e8ce  Make createUd(a)fOperator methods package-private.
     add 1466db9  Merge pull request #13309 from ibzib/fngroup
     add 243e5df  [BEAM-11391] Improve Nexmark Kafka support to track watermarks using commit time, allow specifying the number of partitions instead of requiring launcher to have access to kafka cluster, and removing unneeded deserialization of keys.
     add bfdb3e5  Merge pull request #13472: [BEAM-11391] Improve Nexmark Kafka support to track
     add 164a7b6  [BEAM-9426] Add JVM properties to JavaJarJobServer. (#13581)
     add e9fa621  Release bundle processor when any exceptions during processing.
     add 060b083  Merge pull request #13568 from [BEAM-3245] Release bundle processor when any exceptions during processing.
     add b1fcfe9  [BEAM-10925] Add UdfProvider interface.
     add e4ee5f5  Merge pull request #13587 from ibzib/udf-provider
     add c679de6  [BEAM-9637] Add all runners to Python --runner help text.
     add 0350f2e  Merge pull request #13584 from ibzib/BEAM-9637
     add 926c649  Fix typo in join explanation
     add 44f198d  Merge pull request #13577: Fix typo in join explanation
     add 8a9a259  Fix regex in Kotlin MinimalWordCount example. (#13570)
     add 6171a36  [BEAM-11456] Fix flaky table provider tests (Kafka and Pubsub) (#13542)
     add c2e07fb  [BEAM-10925] Convert ScalarFn to Method. (#13593)
     add a929141  [BEAM-11415][BEAM-11416] Move Go ValidatesRunner tests and add filtering (#13518)
     add 256dafd  [BEAM-11515] Add check for missing authors, add missing authors (#13599)
     add 98ef617  Add 2.28.0 section to CHANGES.md (#13610)
     add c17270b  [BEAM-11159] Use official GCP client in TestPubSubSignal (#13598)
     add fbeb28e  [BEAM-11515] Minor release guide updates (#13609)
     add 30a1baf  [BEAM-11522] Revert "Optimizes extract_output for 1 element accumulator case." (#13612)
     add 0e8725c  Bump Gradle to 6.7.1
     add 5c755ea  Merge pull request #13414: [BEAM-11523] Bump Gradle to 6.7.1
     add feb2b84  Remove usages of WriteStringsToPubSub in examples
     add b0e3d36  Merge pull request #13615 from [BEAM-11524] Remove usages of WriteStringsToPubSub in examples
     add 38eeec9  Merge pull request 2.27.0 from apache/master
     add e4c3c94  Move ParquetIO Changes to 2.28.0
     add 0d344ba  Make passthroughfunction singleton by making the class  with private constructor.
     add 8149158  Merge pull request #13616: [BEAM-11460] Support reading Parquet files with unknown schema
     add 98ee1f1  Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.
     add b6243e7  Merge pull request #13592 from [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.
     add 3256be5  Fix for the case when truststoreLocation and keystoreLocation are not specified
     add 06f8853  Fix nullable coders casting
     add a751c11  Merge pull request #13608 from Fix for Kafka To PubSub Example
     add d61a1fa  [BEAM-11517] Enable test_file_loads on Dataflow
     add e9a7b44  Merge pull request #13601 from [BEAM-11517] Fix test_file_loads on Dataflow
     add 593b576  [BEAM-11531] Restrict pandas version to <1.2.0 (#13623)
     add 8b050a3  [BEAM-10925] Change UserFunctionDefinitions into an AutoValue and Builder class.
     add 356e2ae  Merge pull request #13625 from ibzib/ufd-builder
     add 9f234d6  [BEAM-9980] Combine job_PostCommit_Python*.groovy files into one (#13461)
     add b8f273e  [BEAM-9980] use constant for job_Inventory.groovy and do not use hardcoded python versions (#13460)
     add 8df07b6  [BEAM-11524] Change WriteStringsToPubsub to use WriteToPubsub. (#13614)
     add 2ceef2d  [BEAM-9615] Initial Custom Schema Coder Support (#13611)
     add 894231f  [BEAM-10925] Default to empty maps in UserFunctionDefinitions.Builder.
     add 7ef8bd1  Merge pull request #13628 from ibzib/ufd-builder
     add d84699e  Oxford comma
     add ba305fa  Merge pull request #13631 from elharo/patch-3
     add 4827bea  [BEAM-11537] Fix TestPubSubSignal.waitForStart (#13632)
     add 9de4ea4  [BEAM-11482] Thrift support for KafkaTableProvider (#13572)
     add 4364cc8  [BEAM-10986] Fix for update shadow jar plugin.  (#13586)
     add 0c84a51  BEAM-11536. Test "beam:window_fn:serialized_java:v1" in WindowStrategyTranslation
     add 5e17b69  Merge pull request #13630 from amaliujia/BEAM-11536
     add fe0dfa1  [BEAM-11530] Consider all kinds of annotatios on schema creation (#13622)
     add cec16b6  [BEAM-11532] Fix edge case in merge where left_on and right_on contain equivalent column names (#13634)
     add 9bd4a42  [BEAM-9187] Address flake in loadBalancesBundles (#13448)
     add 48673e8  [BEAM-11533] Add logic to convert Beam schema to DataCatalog schema to SchemaUtils (#13588)
     add a7301b7  [BEAM-11527] Add builder parameter to allow user defined Hadoop ReadSupport flags in Hadoop Configuration.
     add ff5a094  Consistency improvements and other fixes
     add cf74313  Merge pull request #13619: [BEAM-11527] Allow user defined Hadoop ReadSupport flags for ParquetReader
     add 1a617be  Add Beam schema support for ParquetIO reads.
     add 2f205f1  Merge pull request #13639: [BEAM-11526] Add Beam schema support for ParquetIO reads
     add 4165016  [BEAM-11457] Add option to skip key-value clone (#13543)
     add 4370926  [BEAM-10234] Create ApproximateDistinct using HLL Impl
     add 47d6ab5  Merge pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl
     add c7d80a8  Revert "[BEAM-10234] Create ApproximateDistinct using HLL Impl"
     add da1113b  Merge pull request #13653 from apache/revert-12973-hll [BEAM-10324] Reverts apache/beam#12973 Create ApproximateDistinct using HLL Impl
     add cd2d208  Replace . with ;
     add e28cb28  Merge pull request #13659: Replace . with ; in schema programming guide snippet
     add 34a8fcd  [BEAM-11526] Cleanup Code for Schema support (#13646)
     add 47ca61a  [BEAM-10324] Create ApproximateDistinct using HLL Impl (#13655)
     add 3f8a4ff  [BEAM-8829] only drop event_timestamp when it exists (#13638)
     add f120eb7  [BEAM-9541] Add flink_versions to gradle.properties.
     add 7eacb40  [BEAM-9541] All Gradle tasks use latest Flink version.
     add 08706df  Disable Flink classloader leak check when using local execution mode.
     add e12a0d8  [BEAM-11570] Comment with link to context.
     add 9b85af0  Merge pull request #13633 from ibzib/latest-flink
     add 9f9dcd6  [BEAM-9980] do not hardcode Python version for dataflow validate runner tests and make it switchable (#13330)
     add 9276a8c  Copying Github action into our own repository to comply with Apache security requirements
     add b2ad4dd  Merge pull request #13670 from [BEAM-11569] Clone ad-m/github-push-action into Beam repository to comply with Apache security requirements
     add 9ebdc8a  replace Assert.asserThat with MacherAssert.assertThat across entire codebase
     add 3915c3f  [BEAM-11572] Add lint rule to avoid Assert.assertThat usage
     add 9df5db9  Fix Assertion in Nexmark (non-test) class.
     add 60834ae  Merge pull request #13673 from anantdamle/beam_11572_fix_assertThat
     add c72fac9  Fix sdk.properties caching, remove timestamp.
     add d3422f3  Merge pull request #13652 from runningcode/no/remove-timestamp
     add 686b1ff  [BEAM-11569] Building wheels with airflow-clone of github-push action
     add 0183cdb2 Merge pull request #13677 from [BEAM-11569] Building wheels with airflow-clone of github-push action
     add 6a21b06  Fix for Java container release
     add 26d27c8  Merge pull request #13604 from pabloem/fixup-jv811
     add 3b12ce0  [BEAM-11569] Fixing cancel workflows
     add 26c2466  Merge pull request #13679 from [BEAM-11569] Fixing cancel workflows
     add d8966d6  [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework. (#13678)
     add 065ed7c  [BEAM-11415] Adds documentation to integration package. (#13683)
     add d42c67c  [BEAM-11580] Add disk_size_gb flag for Go SDK. (#13693)
     add 854ed5b  [BEAM-10983] Add getting started from Spark page
     add c484e68  Add "From Apache Spark" entry to side menu
     add 17aed03  Add links to transform catalog
     add eaff2e9  Merge branch 'master' of github.com:apache/beam into coming-from-spark
     add 26a2a04  Add more links
     add 2f39799  Add RDD/DataFrame clarification
     add a751df8  Merge branch 'master' of github.com:apache/beam into coming-from-spark
     add 6c338ce  Add link to from-spark page
     add a3406b3  Added extra explanations
     add 67af84a  Merge branch 'master' of github.com:apache/beam into coming-from-spark
     add 9557470  Fixed typos + removed italics
     add 2ad2854  Merge pull request #12963 from davidcavazos/coming-from-spark
     add 96a659a  Clarify some assertions (#13515)
     add 36bf02c  [BEAM-11504] Clean up direct runner parallelism section.
     add d72b5d9  Merge pull request #13589 from ibzib/BEAM-11504
     add 2239b3a  [BEAM-11419] Update Flink supported versions in the website
     add c5badae  Merge pull request #13620 from iemejia/website-flink-versions
     add 41c3811  Revert "[BEAM-11474] Track transform processing thread in Java SDK harness and set log entry field"
     add 6af9d75  Merge pull request #13696 from y1chi/revert-13533-test_logging
     add 93c2bd8  [BEAM-1833] Give transforms more control over the names of their inputs and outputs.
     add 7e8fa4e  mypy
     add 74ec609  Merge pull request #13672 [BEAM-1833] Give transforms more control over the names of their inputs and outputs.
     add 587a3ff  [BEAM-11582] Merge website/.gitignore into .gitignore (#13697)
     add 2e95a20  Correct the license in website package.json file
     add df74d74  Merge pull request #13698 from apache/aaltay-patch-1
     add 078b24d  [BEAM-11543] Increase Spanner streaming read timeout to 2 hours
     add f0db6e7  Merge pull request #13640: [BEAM-11543] Increase Spanner streaming read timeout to 2 hours
     add 995c776  Remove unused endOfStreamMonitor from nexmark
     add b0b8ffd  Also rename metric
     add 38464d1  Merge pull request #13435 from rHermes/remove-unused-monitor-in-nexmark
     add d2f76f9  [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8 (#13410)
     add 3fa9846  [BEAM-11579] Specify flink_job_server_jar instead of using default.
     add 4e83ac5  Merge pull request #13682 from ibzib/BEAM-11579
     add c9f868a  [BEAM-11508] Set sdkHarnessContainerImages in workerpool configuration for Dataflow runner v2
     add ac71556  add utility function to check runner v2, set single-core flag for python
     add a3fef8b  Merge pull request #13605 from ihji/BEAM-11508
     add b378764  [BEAM-11327] Update JulHandler with Charset.defaultCharset() (#13701)
     add 6933486  Adding website update for release 2.27.0
     add ec36893  Merge pull request #13602 from pabloem/2_27_0_website_update
     add ff40901  Fixing script for Java8/Java11 docker container push
     add fbaf292  Merge pull request #13703 from Fixing script for Java8/Java11 docker container push
     add dd12a85  Add 2.27.0 blog post, and update CHANGES.md
     add 2888c0b  Merge branch 'master' into blog-2.27.0
     add 922d7d7  Merge pull request #13603 from pabloem/blog-2.27.0
     add f8fe5a6  Update the blog date for 2.27
     add 868a888  Merge pull request #13708 from apache/aaltay-patch-1
     add 9e7535c  [BEAM-11571] Support Conversion to GenericRecords in Convert.to transform
     add bfee142  [BEAM-11571] Don't transform if input and output types are equal on Convert transform
     add 45ff8df  Merge pull request #13671: [BEAM-11571] Support Conversion to GenericRecords in Convert.to transform
     add 4c51121  [BEAM-11584] Upgrade junit to version 4.13.1
     add 0994dfe  Merge pull request #13700: [BEAM-11584] Upgrade junit to version 4.13.1
     add 20cf66a  [BEAM-8493] Add test to standard double coder (#13713)
     add 5f3a86c  [BEAM-11588] Fix the dependency (#13712)
     add bf814cc  Update Dataflow worker container image version.
     add 15d282b  Merge pull request #13699: Update Dataflow worker harness container image version.
     add 1396a70  don't use artifact to output java 3rd party licenses
     add ca207f6  remove java 3pl directory from copy
     add 7f93e42  Merge pull request #13711 from emilymye/fixjavaartifact
     add 51183ac  Add version and sdk_version as inputs to ProcessResources task.
     add 2c63124  Also add docker image to sdk.properties
     add a1d97b8  Fix incorrect input property
     add 0a7edfc  Update sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
     add 75d3864  Merge pull request #13691 from runningcode/no/sdk-properties
     add f46954c  [BEAM-11588] Move forward jupyter-client version.
     add 04e3343  Merge pull request #13721 from KevinGG/BEAM-11588
     add 8e1a8b8  [BEAM-10402] Trigger#subTriggers does not return null
     add 7aa7e13  Merge pull request #13719: [BEAM-10402] Trigger#subTriggers might return null
     add dd71c9a  Update doc/examples: BigQuerySource to ReadFromBigQuery (#13239)
     add 4c8f0b0  [BEAM-11474] Set log entry transform id with best effort
     add 327ec71  Address comment
     add 6c9da02  Merge pull request #13702 from [BEAM-11474] Set log entry transform id with best effort in Java SDK harness
     add 1aca0c7  Note about SQL in Python.
     add 53615fe  Merge pull request #13738 Update docs wrt SQL in Python.
     new 7d0d220  Merge remote-tracking branch 'origin/master' into website-revamp

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/workflows/build_wheels.yml                 |   2 +-
 .github/workflows/cancel.yml                       |   2 +-
 .gitignore                                         |  10 +
 .test-infra/jenkins/README.md                      |   9 +-
 .test-infra/jenkins/job_Inventory.groovy           |  15 +-
 .../jenkins/job_LoadTests_Combine_Flink_Go.groovy  |   2 +-
 ...k_Go.groovy => job_LoadTests_Combine_Go.groovy} |  85 +-
 .../jenkins/job_LoadTests_GBK_Flink_Go.groovy      |  19 +-
 .test-infra/jenkins/job_LoadTests_GBK_Go.groovy    | 211 +++++
 .test-infra/jenkins/job_LoadTests_ParDo_Go.groovy  | 145 ++++
 .../job_LoadTests_SideInput_Flink_Go.groovy        | 103 +++
 .../jenkins/job_LoadTests_SideInput_Go.groovy      |  94 +++
 .test-infra/jenkins/job_LoadTests_coGBK_Go.groovy  | 169 ++++
 ...ython36.groovy => job_PostCommit_Python.groovy} |  39 +-
 .test-infra/jenkins/job_PostCommit_Python37.groovy |  44 -
 .test-infra/jenkins/job_PostCommit_Python38.groovy |  44 -
 ...stCommit_Python_ValidatesRunner_Dataflow.groovy |   8 +-
 ...ommit_Python_ValidatesRunner_Dataflow_V2.groovy |   9 +-
 .../jenkins/job_Publish_SDK_Image_Snapshots.groovy |   8 +-
 .../perftests_metrics/SideInput_Load_Tests.json    |   2 +-
 CHANGES.md                                         |  51 +-
 build.gradle                                       |   9 -
 buildSrc/build.gradle                              |   4 +-
 .../org/apache/beam/gradle/BeamModulePlugin.groovy |  31 +-
 examples/java/build.gradle                         |  11 +-
 .../org/apache/beam/examples/complete/README.md    |   5 +
 .../kafkatopubsub/KafkaPubsubConstants.java        |  15 +-
 .../complete/kafkatopubsub/KafkaToPubsub.java      | 235 ++++++
 .../beam/examples/complete/kafkatopubsub/README.md | 200 +++++
 .../complete/kafkatopubsub/avro/AvroDataClass.java |  63 ++
 .../avro/AvroDataClassKafkaAvroDeserializer.java   |  27 +-
 .../complete/kafkatopubsub/avro/package-info.java  |  10 +-
 .../kafka/consumer/SslConsumerFactoryFn.java       | 135 ++++
 .../kafkatopubsub/kafka/consumer/Utils.java        | 165 ++++
 .../kafkatopubsub/kafka/consumer/package-info.java |  10 +-
 .../options/KafkaToPubsubOptions.java              |  90 +++
 .../kafkatopubsub/options/package-info.java        |  10 +-
 .../complete/kafkatopubsub/package-info.java       |  10 +-
 .../kafkatopubsub/transforms/FormatTransform.java  | 128 +++
 .../kafkatopubsub/transforms/package-info.java     |  10 +-
 .../examples/complete/game/LeaderBoardTest.java    |   2 +-
 .../complete/kafkatopubsub/KafkaToPubsubTest.java  |  90 +++
 .../cookbook/CombinePerKeyExamplesTest.java        |  13 +-
 examples/kotlin/build.gradle                       |   2 +-
 .../beam/examples/kotlin/MinimalWordCount.kt       |   2 +-
 gradle.properties                                  |   7 +-
 gradle/wrapper/gradle-wrapper.properties           |   2 +-
 .../beam/learning/katas/util/ContainsKvs.java      |   2 +-
 .../pipeline/src/main/proto/beam_runner_api.proto  |  18 +
 .../src/main/scripts/build_release_candidate.sh    |   6 +-
 .../src/main/scripts/preparation_before_release.sh |   2 +
 release/src/main/scripts/publish_docker_images.sh  |   4 +-
 .../core/construction/CoderTranslationTest.java    |   2 +-
 .../runners/core/construction/CommonCoderTest.java |   2 +-
 .../CreatePCollectionViewTranslationTest.java      |   2 +-
 .../DeduplicatedFlattenFactoryTest.java            |   2 +-
 .../construction/DefaultArtifactResolverTest.java  |   2 +-
 .../EmptyFlattenAsCreateFactoryTest.java           |   2 +-
 .../core/construction/EnvironmentsTest.java        |   2 +-
 .../ExecutableStageTranslationTest.java            |   2 +-
 .../construction/ForwardingPTransformTest.java     |   2 +-
 .../construction/GroupByKeyTranslationTest.java    |   2 +-
 .../runners/core/construction/ModelCodersTest.java |   2 +-
 .../core/construction/MorePipelineTest.java        |   2 +-
 .../construction/PCollectionTranslationTest.java   |   2 +-
 .../core/construction/PTransformMatchersTest.java  |   2 +-
 .../construction/PTransformReplacementsTest.java   |   2 +-
 .../construction/PTransformTranslationTest.java    |   2 +-
 .../core/construction/ParDoTranslationTest.java    |   2 +-
 .../PipelineOptionsTranslationTest.java            |   2 +-
 .../core/construction/PipelineTranslationTest.java |   2 +-
 .../core/construction/ReadTranslationTest.java     |   2 +-
 .../construction/RehydratedComponentsTest.java     |   2 +-
 .../core/construction/ReplacementOutputsTest.java  |   2 +-
 .../construction/ReshuffleTranslationTest.java     |   2 +-
 .../core/construction/SchemaTranslationTest.java   |   2 +-
 .../core/construction/SdkComponentsTest.java       |   2 +-
 .../SingleInputOutputOverrideFactoryTest.java      |   2 +-
 .../construction/TestStreamTranslationTest.java    |   2 +-
 .../core/construction/TransformInputsTest.java     |   2 +-
 .../core/construction/TriggerTranslationTest.java  |   2 +-
 .../core/construction/UnconsumedReadsTest.java     |   2 +-
 .../core/construction/ValidateRunnerXlangTest.java |   2 +-
 .../WindowingStrategyTranslationTest.java          | 161 +++-
 .../construction/WriteFilesTranslationTest.java    |   2 +-
 .../construction/graph/ExecutableStageTest.java    |   2 +-
 .../core/construction/graph/FusedPipelineTest.java |   2 +-
 .../graph/GreedyPipelineFuserTest.java             |   2 +-
 .../construction/graph/GreedyStageFuserTest.java   |   2 +-
 .../graph/ImmutableExecutableStageTest.java        |   2 +-
 .../core/construction/graph/NetworksTest.java      |   2 +-
 .../construction/graph/OutputDeduplicatorTest.java |   2 +-
 .../construction/graph/ProtoOverridesTest.java     |   2 +-
 .../construction/graph/QueryablePipelineTest.java  |   2 +-
 .../core/metrics/ExecutionStateTracker.java        |  20 +-
 .../runners/core/InMemoryStateInternalsTest.java   |   2 +-
 .../runners/core/InMemoryTimerInternalsTest.java   |   2 +-
 .../core/LateDataDroppingDoFnRunnerTest.java       |   2 +-
 .../beam/runners/core/LateDataUtilsTest.java       |   2 +-
 ...BoundedSplittableProcessElementInvokerTest.java |   2 +-
 .../beam/runners/core/ReduceFnRunnerTest.java      |   2 +-
 .../beam/runners/core/SideInputHandlerTest.java    |   2 +-
 .../beam/runners/core/SimpleDoFnRunnerTest.java    |   2 +-
 .../SimplePushbackSideInputDoFnRunnerTest.java     |   2 +-
 .../runners/core/SplittableParDoProcessFnTest.java |   2 +-
 .../beam/runners/core/StateNamespacesTest.java     |   2 +-
 .../beam/runners/core/TimerInternalsTest.java      |   2 +-
 .../beam/runners/core/WindowMatchersTest.java      |   2 +-
 .../beam/runners/core/metrics/CounterCellTest.java |   2 +-
 .../beam/runners/core/metrics/DirtyStateTest.java  |   2 +-
 .../runners/core/metrics/DistributionCellTest.java |   2 +-
 .../core/metrics/ExecutionStateSamplerTest.java    |   2 +-
 .../core/metrics/ExecutionStateTrackerTest.java    |   2 +-
 .../beam/runners/core/metrics/GaugeCellTest.java   |   2 +-
 .../core/metrics/MetricsContainerImplTest.java     |   2 +-
 .../core/metrics/MetricsContainerStepMapTest.java  |   2 +-
 .../beam/runners/core/metrics/MetricsMapTest.java  |   2 +-
 .../runners/core/metrics/MetricsPusherTest.java    |   2 +-
 .../core/metrics/MonitoringInfoConstantsTest.java  |   2 +-
 .../core/metrics/SimpleStateRegistryTest.java      |   2 +-
 .../triggers/AfterWatermarkStateMachineTest.java   |   2 +-
 .../core/triggers/FinishedTriggersBitSetTest.java  |   2 +-
 .../core/triggers/FinishedTriggersSetTest.java     |   2 +-
 .../core/triggers/NeverStateMachineTest.java       |   2 +-
 .../core/triggers/TriggerStateMachinesTest.java    |   2 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java    |   2 +-
 .../runners/direct/CloningBundleFactoryTest.java   |   2 +-
 .../beam/runners/direct/CommittedResultTest.java   |   2 +-
 .../CopyOnAccessInMemoryStateInternalsTest.java    |   2 +-
 .../runners/direct/DirectGraphVisitorTest.java     |   2 +-
 .../DirectGroupByKeyOverrideFactoryTest.java       |   2 +-
 .../beam/runners/direct/DirectMetricsTest.java     |   2 +-
 .../runners/direct/DirectRunnerApiSurfaceTest.java |   2 +-
 .../beam/runners/direct/DirectRunnerTest.java      |   2 +-
 .../runners/direct/DirectTimerInternalsTest.java   |   2 +-
 .../direct/DirectTransformExecutorTest.java        |   2 +-
 ...cycleManagerRemovingTransformEvaluatorTest.java |   2 +-
 .../runners/direct/DoFnLifecycleManagerTest.java   |   2 +-
 .../beam/runners/direct/EvaluationContextTest.java |   2 +-
 .../direct/FlattenEvaluatorFactoryTest.java        |   2 +-
 .../direct/GroupByKeyEvaluatorFactoryTest.java     |   2 +-
 .../direct/GroupByKeyOnlyEvaluatorFactoryTest.java |   2 +-
 .../ImmutabilityCheckingBundleFactoryTest.java     |   2 +-
 .../direct/ImmutableListBundleFactoryTest.java     |   2 +-
 .../direct/ImpulseEvaluatorFactoryTest.java        |   2 +-
 .../direct/KeyedPValueTrackingVisitorTest.java     |   2 +-
 .../beam/runners/direct/MultiStepCombineTest.java  |   2 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java    |   2 +-
 .../runners/direct/SideInputContainerTest.java     |   2 +-
 .../direct/StatefulParDoEvaluatorFactoryTest.java  |   2 +-
 .../runners/direct/StepTransformResultTest.java    |   2 +-
 .../direct/TestStreamEvaluatorFactoryTest.java     |   2 +-
 .../direct/UnboundedReadDeduplicatorTest.java      |   2 +-
 .../direct/UnboundedReadEvaluatorFactoryTest.java  |   2 +-
 .../runners/direct/ViewEvaluatorFactoryTest.java   |   2 +-
 .../direct/WatermarkCallbackExecutorTest.java      |   2 +-
 .../beam/runners/direct/WatermarkManagerTest.java  |   2 +-
 .../runners/direct/WindowEvaluatorFactoryTest.java |   2 +-
 .../direct/WriteWithShardingFactoryTest.java       |   2 +-
 .../runners/flink/streaming/StreamSources.java     |  13 +
 {sdks/go => runners/flink/1.12}/build.gradle       |  38 +-
 .../job-server-container/build.gradle              |   0
 .../flink/1.12/job-server}/build.gradle            |  34 +-
 .../streaming/AbstractStreamOperatorCompat.java    |  53 ++
 .../runners/flink/streaming/StreamSources.java     |  14 +
 .../runners/flink/streaming/StreamSources.java     |  13 +
 .../runners/flink/streaming/StreamSources.java     |  13 +
 runners/flink/flink_runner.gradle                  |  19 +-
 .../runners/flink/FlinkExecutionEnvironments.java  |  12 +
 .../flink/FlinkPortableClientEntryPoint.java       |   4 +-
 .../FlinkStreamingPortablePipelineTranslator.java  |  27 +-
 .../flink/FlinkStreamingTransformTranslators.java  | 215 +++--
 .../flink/FlinkStreamingTranslationContext.java    |  24 +-
 .../wrappers/streaming/DoFnOperator.java           |  77 +-
 .../streaming/KeyedPushedBackElementsHandler.java  |   5 +-
 .../streaming/SingletonKeyedWorkItemCoder.java     |  13 +-
 .../wrappers/streaming/WorkItemKeySelector.java    |  13 +-
 .../streaming/state/FlinkStateInternals.java       |   4 +-
 .../FlinkStreamingPipelineTranslatorTest.java      | 109 +++
 .../FlinkStreamingTransformTranslatorsTest.java    |   7 +-
 .../runners/flink/FlinkTransformOverridesTest.java |   2 +-
 .../wrappers/streaming/DedupingOperatorTest.java   |   2 +-
 .../wrappers/streaming/DoFnOperatorTest.java       | 315 ++++++--
 .../wrappers/streaming/WindowDoFnOperatorTest.java |   5 +-
 runners/google-cloud-dataflow-java/build.gradle    |   2 +-
 .../beam/runners/dataflow/DataflowRunner.java      |  53 +-
 .../options/DataflowPipelineDebugOptions.java      |  10 +
 .../dataflow/BatchStatefulParDoOverridesTest.java  |   2 +-
 .../runners/dataflow/BatchViewOverridesTest.java   |   2 +-
 .../dataflow/DataflowPTransformMatchersTest.java   |   2 +-
 .../dataflow/DataflowPipelineTranslatorTest.java   |   2 +-
 .../runners/dataflow/DataflowRunnerInfoTest.java   |   2 +-
 .../beam/runners/dataflow/DataflowRunnerTest.java  |  56 ++
 .../dataflow/PrimitiveParDoSingleFactoryTest.java  |   2 +-
 .../runners/dataflow/TestDataflowRunnerTest.java   |   2 +-
 .../options/DataflowPipelineDebugOptionsTest.java  |   2 +-
 .../options/DataflowProfilingOptionsTest.java      |   2 +-
 .../runners/dataflow/util/PackageUtilTest.java     |   2 +-
 .../beam/runners/dataflow/util/StructsTest.java    |  25 +-
 .../beam/runners/dataflow/worker/ReaderCache.java  |  38 +-
 .../dataflow/worker/StreamingDataflowWorker.java   |  61 +-
 .../worker/fn/data/BeamFnDataGrpcService.java      |  14 +-
 .../logging/DataflowWorkerLoggingInitializer.java  |  14 +-
 .../JulHandlerPrintStreamAdapterFactory.java       |  15 +-
 .../dataflow/harness/test/TestStreamsTest.java     |   2 +-
 .../dataflow/worker/AvroByteReaderFactoryTest.java |   5 +-
 .../dataflow/worker/AvroByteSinkFactoryTest.java   |   3 +-
 .../dataflow/worker/BatchDataflowWorkerTest.java   |   2 +-
 .../worker/BatchModeExecutionContextTest.java      |   2 +-
 .../dataflow/worker/ConcatReaderFactoryTest.java   |   2 +-
 .../runners/dataflow/worker/ConcatReaderTest.java  |   2 +-
 .../ContextActivationObserverRegistryTest.java     |   2 +-
 ...CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java |   2 +-
 .../DataflowElementExecutionTrackerTest.java       |   2 +-
 .../worker/DataflowExecutionStateTrackerTest.java  |   2 +-
 .../worker/DataflowSideInputReadCounterTest.java   |   2 +-
 .../worker/DataflowWorkerHarnessHelperTest.java    |   2 +-
 .../dataflow/worker/DefaultParDoFnFactoryTest.java |   2 +-
 .../dataflow/worker/DeltaCounterCellTest.java      |   2 +-
 .../dataflow/worker/DoFnInstanceManagersTest.java  |   2 +-
 .../dataflow/worker/FakeWindmillServer.java        |  14 +-
 .../runners/dataflow/worker/FilepatternsTest.java  |   2 +-
 .../GroupAlsoByWindowParDoFnFactoryTest.java       |   2 +-
 .../dataflow/worker/InMemoryReaderFactoryTest.java |   3 +-
 .../IntrinsicMapTaskExecutorFactoryTest.java       |   2 +-
 .../worker/IntrinsicMapTaskExecutorTest.java       |   2 +-
 .../runners/dataflow/worker/IsmFormatTest.java     |   2 +-
 .../dataflow/worker/IsmSideInputReaderTest.java    |   2 +-
 .../dataflow/worker/LogRecordMatcherTest.java      |   2 +-
 .../beam/runners/dataflow/worker/LogSaverTest.java |   2 +-
 ...tContextActivationObserverRegistrationTest.java |   2 +-
 .../worker/PairWithConstantKeyDoFnFactoryTest.java |   2 +-
 .../worker/PartialGroupByKeyParDoFnsTest.java      |   2 +-
 .../runners/dataflow/worker/ReaderCacheTest.java   |   6 +-
 .../runners/dataflow/worker/ReaderFactoryTest.java |   6 +-
 ...ReifyTimestampAndWindowsParDoFnFactoryTest.java |   2 +-
 ...essCoderCloudObjectTranslatorRegistrarTest.java |   2 +-
 .../dataflow/worker/ShuffleReaderFactoryTest.java  |   3 +-
 .../dataflow/worker/ShuffleSinkFactoryTest.java    |   3 +-
 .../runners/dataflow/worker/SimpleParDoFnTest.java |   2 +-
 .../runners/dataflow/worker/SinkRegistryTest.java  |   5 +-
 .../worker/SourceOperationExecutorFactoryTest.java |   2 +-
 .../runners/dataflow/worker/StateFetcherTest.java  |   2 +-
 .../worker/StreamingDataflowWorkerTest.java        |  10 +-
 .../worker/StreamingGroupAlsoByWindowFnsTest.java  |   2 +-
 ...reamingGroupAlsoByWindowsReshuffleDoFnTest.java |   2 +-
 .../worker/StreamingModeExecutionContextTest.java  |   5 +-
 ...eamingPCollectionViewWriterDoFnFactoryTest.java |   2 +-
 .../worker/StreamingSideInputDoFnRunnerTest.java   |   2 +-
 .../worker/StreamingSideInputFetcherTest.java      |   2 +-
 .../worker/StreamingStepMetricsContainerTest.java  |   2 +-
 .../ToIsmRecordForMultimapDoFnFactoryTest.java     |   2 +-
 .../dataflow/worker/UserParDoFnFactoryTest.java    |   2 +-
 .../dataflow/worker/ValuesDoFnFactoryTest.java     |   2 +-
 .../dataflow/worker/WindmillKeyedWorkItemTest.java |   2 +-
 .../worker/WindmillStateInternalsTest.java         |   2 +-
 .../dataflow/worker/WindmillStateReaderTest.java   |   2 +-
 .../worker/WindmillTimerInternalsTest.java         |   2 +-
 .../dataflow/worker/WorkItemStatusClientTest.java  |   2 +-
 .../dataflow/worker/WorkerCustomSourcesTest.java   |   2 +-
 .../dataflow/worker/counters/CounterSetTest.java   |   2 +-
 .../fn/control/BeamFnMapTaskExecutorTest.java      |   2 +-
 .../RegisterAndProcessBundleOperationTest.java     |   2 +-
 .../worker/fn/data/BeamFnDataGrpcServiceTest.java  |   2 +-
 .../fn/data/RemoteGrpcPortReadOperationTest.java   |   2 +-
 .../fn/data/RemoteGrpcPortWriteOperationTest.java  |   2 +-
 .../fn/logging/BeamFnLoggingServiceTest.java       |   2 +-
 .../fn/stream/ServerStreamObserverFactoryTest.java |   2 +-
 .../graph/CloneAmbiguousFlattensFunctionTest.java  |   2 +-
 .../CreateRegisterFnOperationFunctionTest.java     |   2 +-
 .../graph/DeduceNodeLocationsFunctionTest.java     |   2 +-
 ...tFetchAndFilterStreamingSideInputNodesTest.java |   2 +-
 .../worker/graph/MapTaskToNetworkFunctionTest.java |   2 +-
 .../dataflow/worker/graph/NetworksTest.java        |   2 +-
 .../RemoveFlattenInstructionsFunctionTest.java     |   2 +-
 .../ReplacePgbkWithPrecombineFunctionTest.java     |   2 +-
 .../DataflowWorkerLoggingInitializerTest.java      |   2 +-
 .../JulHandlerPrintStreamAdapterFactoryTest.java   |  16 +-
 .../worker/profiler/ScopedProfilerTest.java        |   2 +-
 .../dataflow/worker/status/ThreadzServletTest.java |   2 +-
 .../worker/status/WorkerStatusPagesTest.java       |   2 +-
 .../worker/util/BatchGroupAlsoByWindowFnsTest.java |   2 +-
 .../BatchGroupAlsoByWindowReshuffleDoFnTest.java   |   2 +-
 .../worker/util/GroupAlsoByWindowProperties.java   |   2 +-
 .../dataflow/worker/util/MemoryMonitorTest.java    |   2 +-
 .../worker/util/ScalableBloomFilterTest.java       |   2 +-
 .../dataflow/worker/util/TimerOrElementTest.java   |   2 +-
 .../worker/BatchingShuffleEntryReaderTest.java     |   2 +-
 .../worker/CachingShuffleBatchReaderTest.java      |   2 +-
 .../util/common/worker/FlattenOperationTest.java   |   5 +-
 .../util/common/worker/GroupingTablesTest.java     |   2 +-
 .../util/common/worker/MapTaskExecutorTest.java    |   2 +-
 .../util/common/worker/OutputReceiverTest.java     |   5 +-
 .../util/common/worker/ParDoOperationTest.java     |   2 +-
 .../util/common/worker/ReadOperationTest.java      |   2 +-
 .../util/common/worker/ShuffleEntryTest.java       |   2 +-
 .../util/common/worker/WriteOperationTest.java     |   2 +-
 .../runners/fnexecution/data/GrpcDataService.java  |   6 +-
 .../runners/fnexecution/ServerFactoryTest.java     |   2 +-
 .../control/DefaultJobBundleFactoryTest.java       |  62 +-
 .../control/FnApiControlClientPoolServiceTest.java |   2 +-
 .../control/FnApiControlClientTest.java            |   2 +-
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |   2 +-
 .../fnexecution/data/GrpcDataServiceTest.java      |   2 +-
 .../environment/ProcessEnvironmentFactoryTest.java |   2 +-
 .../environment/ProcessManagerTest.java            |   8 +-
 .../environment/RemoteEnvironmentTest.java         |   2 +-
 .../logging/GrpcLoggingServiceTest.java            |   2 +-
 .../StaticGrpcProvisionServiceTest.java            |   2 +-
 .../fnexecution/state/GrpcStateServiceTest.java    |   2 +-
 .../translation/PipelineTranslatorUtilsTest.java   |   2 +-
 .../beam/runners/local/StructuralKeyTest.java      |  18 +-
 runners/portability/java/build.gradle              |  24 -
 runners/samza/build.gradle                         |  17 +-
 .../beam/runners/samza/SamzaJobInvocation.java     | 114 +++
 .../beam/runners/samza/SamzaJobServerDriver.java   |  60 +-
 .../beam/runners/samza/SamzaPipelineOptions.java   |  22 +-
 .../samza/SamzaPipelineOptionsValidator.java       |  38 +-
 .../samza/SamzaPortablePipelineOptions.java        |  13 +
 .../runners/samza/SamzaRunnerOverrideConfigs.java  |  15 +
 .../samza/adapter/UnboundedSourceSystem.java       |  39 +-
 .../samza/container/BeamContainerRunner.java       |   6 +-
 .../samza/container/BeamJobCoordinatorRunner.java  |  78 ++
 ...inerCfgFactory.java => ContainerCfgLoader.java} |  18 +-
 .../ContainerCfgLoaderFactory.java}                |  16 +-
 .../beam/runners/samza/runtime/BundleManager.java  | 349 ++++++++
 .../apache/beam/runners/samza/runtime/DoFnOp.java  | 277 ++++---
 .../runners/samza/runtime/FutureCollector.java     |  60 ++
 .../beam/runners/samza/runtime/GroupByKeyOp.java   |  11 +-
 .../beam/runners/samza/runtime/KeyedTimerData.java |   1 +
 .../beam/runners/samza/runtime/OpAdapter.java      |  48 +-
 .../beam/runners/samza/runtime/OpEmitter.java      |   5 +
 .../samza/runtime/OutputManagerFactory.java        |   5 +
 .../samza/runtime/SamzaStoreStateInternals.java    | 230 ++++--
 .../samza/runtime/SamzaTimerInternalsFactory.java  | 217 ++++-
 .../runners/samza/translation/ConfigBuilder.java   |  77 +-
 .../runners/samza/translation/ConfigContext.java   |   8 +
 .../translation/FlattenPCollectionsTranslator.java |   2 +-
 .../samza/translation/GroupByKeyTranslator.java    |  24 +-
 .../translation/ParDoBoundMultiTranslator.java     |  56 +-
 .../samza/translation/SamzaPipelineTranslator.java |  21 +-
 .../translation/SamzaTestStreamSystemFactory.java  | 179 +++++
 .../translation/SamzaTestStreamTranslator.java     | 100 +++
 .../translation/SplittableParDoTranslators.java    |   6 +-
 .../samza/translation/TranslationContext.java      |  59 +-
 .../samza/translation/WindowAssignTranslator.java  |   4 +-
 .../beam/runners/samza/util/FutureUtils.java       |  50 ++
 .../samza/SamzaPipelineOptionsValidatorTest.java   |  60 ++
 .../samza/adapter/UnboundedSourceSystemTest.java   |  28 +
 .../runners/samza/runtime/BundleManagerTest.java   | 474 +++++++++++
 .../samza/runtime/FutureCollectorImplTest.java     |  92 +++
 .../runners/samza/runtime/KeyedTimerDataTest.java  |  14 +-
 .../runtime/SamzaStoreStateInternalsTest.java      |  38 +-
 .../runtime/SamzaTimerInternalsFactoryTest.java    | 416 +++++++++-
 .../samza/translation/ConfigGeneratorTest.java     |  86 +-
 .../samza/translation/TranslationContextTest.java  |  94 +++
 .../beam/runners/samza/util/FutureUtilsTest.java   | 107 +++
 .../runners/spark/GlobalWatermarkHolderTest.java   |   2 +-
 .../beam/runners/spark/SparkPipelineStateTest.java |   2 +-
 .../runners/spark/SparkRunnerDebuggerTest.java     |   2 +-
 .../metrics/sink/SparkMetricsSinkTest.java         |   2 +-
 .../spark/io/ReaderToIteratorAdapterTest.java      |   2 +-
 .../runners/spark/metrics/SparkBeamMetricTest.java |   2 +-
 .../spark/metrics/SparkMetricsPusherTest.java      |   2 +-
 .../metrics/sink/SparkMetricsSinkTest.java         |   2 +-
 .../metrics/BeamMetricTest.java                    |   2 +-
 .../translation/batch/GroupByKeyTest.java          |   2 +-
 .../translation/streaming/CreateStreamTest.java    |   2 +-
 .../ResumeFromCheckpointStreamingTest.java         |   2 +-
 .../streaming/SparkCoGroupByKeyStreamingTest.java  |   2 +-
 .../streaming/StreamingSourceMetricsTest.java      |   2 +-
 .../streaming/TrackStreamingSourcesTest.java       |   2 +-
 sdks/go/build.gradle                               |   2 +-
 sdks/go/container/build.gradle                     |   2 +
 sdks/go/pkg/beam/core/graph/coder/double_test.go   |  53 ++
 sdks/go/pkg/beam/core/graph/coder/map_test.go      |  11 +-
 sdks/go/pkg/beam/core/graph/coder/row.go           | 337 ++------
 sdks/go/pkg/beam/core/graph/coder/row_decoder.go   | 308 +++++++
 sdks/go/pkg/beam/core/graph/coder/row_encoder.go   | 271 +++++++
 sdks/go/pkg/beam/core/graph/coder/row_test.go      | 488 +++++++++---
 .../pkg/beam/core/graph/coder/testutil/testutil.go | 154 ++++
 .../core/graph/coder/testutil/testutil_test.go     | 201 +++++
 sdks/go/pkg/beam/core/metrics/metrics.go           |  39 +
 sdks/go/pkg/beam/core/runtime/graphx/xlang.go      |   9 +-
 sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go |  38 +-
 sdks/go/pkg/beam/pipeline.go                       |   1 +
 sdks/go/pkg/beam/runners/dataflow/dataflow.go      |   5 +-
 .../beam/runners/dataflow/dataflowlib/execute.go   |  57 +-
 .../pkg/beam/runners/dataflow/dataflowlib/job.go   |   8 +
 .../beam/runners/dataflow/dataflowlib/metrics.go   | 124 +++
 .../runners/dataflow/dataflowlib/metrics_test.go   | 133 ++++
 .../beam/runners/universal/runnerlib/compile.go    |  13 +-
 .../beam/runners/universal/runnerlib/execute.go    |  21 +-
 sdks/go/pkg/beam/runners/universal/universal.go    |   1 +
 sdks/go/pkg/beam/testing/ptest/ptest.go            |  21 +
 sdks/go/test/build.gradle                          |  94 ++-
 sdks/go/test/integration/{ => driver}/driver.go    |   0
 sdks/go/test/integration/integration.go            | 147 ++++
 sdks/go/test/integration/primitives/cogbk_test.go  |  16 +-
 .../go/test/integration/primitives/flatten_test.go |  11 +-
 sdks/go/test/integration/primitives/pardo_test.go  |  16 +-
 .../test/integration/synthetic/synthetic_test.go   |  75 ++
 .../test/integration/wordcount/wordcount_test.go   |   6 +
 sdks/go/test/integration/xlang/xlang_test.go       | 260 ++++++
 sdks/go/test/load/build.gradle                     |   1 +
 sdks/go/test/load/sideinput/sideinput.go           | 100 +++
 sdks/go/test/run_integration_tests.sh              |   2 +-
 ...ation_tests.sh => run_validatesrunner_tests.sh} | 220 +++--
 .../src/main/resources/beam/checkstyle.xml         |  17 +
 .../src/main/resources/beam/suppressions.xml       |   4 +
 sdks/java/container/build.gradle                   |   4 +-
 sdks/java/container/common.gradle                  |  12 +-
 sdks/java/core/build.gradle                        |   5 +-
 .../main/java/org/apache/beam/sdk/io/AvroIO.java   |  38 +-
 .../apache/beam/sdk/io/AvroSchemaIOProvider.java   |   4 +-
 .../beam/sdk/io/GenericRecordWriteConverter.java   |  67 --
 .../src/main/java/org/apache/beam/sdk/io/Read.java |  97 ++-
 .../sdk/schemas/FieldValueTypeInformation.java     |  53 +-
 .../org/apache/beam/sdk/schemas/RowMessages.java   | 132 +++
 .../beam/sdk/schemas/transforms/Convert.java       |   5 +
 .../apache/beam/sdk/schemas/utils/AvroUtils.java   |   4 +-
 .../beam/sdk/schemas/utils/ConvertHelpers.java     |  14 +-
 .../apache/beam/sdk/testing/CoderProperties.java   |   2 +-
 .../apache/beam/sdk/testing/CombineFnTester.java   |   2 +-
 .../java/org/apache/beam/sdk/testing/PAssert.java  |   2 +-
 .../apache/beam/sdk/testing/SourceTestUtils.java   |   2 +-
 .../org/apache/beam/sdk/testing/TestPipeline.java  |  45 ++
 .../apache/beam/sdk/testing/WindowFnTestUtils.java |   2 +-
 .../beam/sdk/transforms/ApproximateUnique.java     |  20 +-
 .../beam/sdk/transforms/windowing/Trigger.java     |   3 +-
 .../resources/org/apache/beam/sdk/sdk.properties   |   5 +-
 .../java/org/apache/beam/sdk/PipelineTest.java     |   2 +-
 .../org/apache/beam/sdk/coders/AvroCoderTest.java  |   2 +-
 .../beam/sdk/coders/BigDecimalCoderTest.java       |   2 +-
 .../beam/sdk/coders/BigEndianIntegerCoderTest.java |   2 +-
 .../beam/sdk/coders/BigEndianLongCoderTest.java    |   2 +-
 .../beam/sdk/coders/BigIntegerCoderTest.java       |   2 +-
 .../apache/beam/sdk/coders/BitSetCoderTest.java    |   2 +-
 .../apache/beam/sdk/coders/ByteArrayCoderTest.java |   2 +-
 .../org/apache/beam/sdk/coders/ByteCoderTest.java  |   2 +-
 .../java/org/apache/beam/sdk/coders/CoderTest.java |   2 +-
 .../beam/sdk/coders/CollectionCoderTest.java       |   2 +-
 .../apache/beam/sdk/coders/DefaultCoderTest.java   |   2 +-
 .../apache/beam/sdk/coders/DelegateCoderTest.java  |   2 +-
 .../apache/beam/sdk/coders/DoubleCoderTest.java    |   2 +-
 .../apache/beam/sdk/coders/DurationCoderTest.java  |   2 +-
 .../org/apache/beam/sdk/coders/FloatCoderTest.java |   2 +-
 .../apache/beam/sdk/coders/InstantCoderTest.java   |   2 +-
 .../apache/beam/sdk/coders/IterableCoderTest.java  |   2 +-
 .../org/apache/beam/sdk/coders/KvCoderTest.java    |   2 +-
 .../org/apache/beam/sdk/coders/ListCoderTest.java  |   2 +-
 .../org/apache/beam/sdk/coders/MapCoderTest.java   |   2 +-
 .../apache/beam/sdk/coders/NullableCoderTest.java  |   2 +-
 .../sdk/coders/PCollectionCustomCoderTest.java     |   2 +-
 .../beam/sdk/coders/SerializableCoderTest.java     |   2 +-
 .../org/apache/beam/sdk/coders/SetCoderTest.java   |   2 +-
 .../beam/sdk/coders/StringDelegateCoderTest.java   |   2 +-
 .../beam/sdk/coders/StringUtf8CoderTest.java       |   2 +-
 .../beam/sdk/coders/StructuredCoderTest.java       |  11 +-
 .../beam/sdk/coders/TextualIntegerCoderTest.java   |   2 +-
 .../apache/beam/sdk/coders/VarIntCoderTest.java    |   2 +-
 .../apache/beam/sdk/coders/VarLongCoderTest.java   |   2 +-
 .../org/apache/beam/sdk/coders/VoidCoderTest.java  |   2 +-
 .../java/org/apache/beam/sdk/io/AvroIOTest.java    |   2 +-
 .../org/apache/beam/sdk/io/AvroSourceTest.java     |   2 +-
 .../sdk/io/BoundedReadFromUnboundedSourceTest.java |   2 +-
 .../apache/beam/sdk/io/CompressedSourceTest.java   |   2 +-
 .../org/apache/beam/sdk/io/CountingSourceTest.java |   2 +-
 .../org/apache/beam/sdk/io/FileBasedSinkTest.java  |   2 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java    |   2 +-
 .../org/apache/beam/sdk/io/FileSystemsTest.java    |   2 +-
 .../beam/sdk/io/LocalFileSystemRegistrarTest.java  |   2 +-
 .../apache/beam/sdk/io/LocalFileSystemTest.java    |   2 +-
 .../test/java/org/apache/beam/sdk/io/ReadTest.java | 130 +++
 .../org/apache/beam/sdk/io/TextIOReadTest.java     |   2 +-
 .../org/apache/beam/sdk/io/TextIOWriteTest.java    |   2 +-
 .../org/apache/beam/sdk/io/WriteFilesTest.java     |   2 +-
 .../io/range/ByteKeyRangeEstimateFractionTest.java |   2 +-
 .../io/range/ByteKeyRangeInterpolateKeyTest.java   |   2 +-
 .../apache/beam/sdk/io/range/ByteKeyRangeTest.java |   2 +-
 .../org/apache/beam/sdk/io/range/ByteKeyTest.java  |   2 +-
 .../org/apache/beam/sdk/metrics/MetricsTest.java   |   2 +-
 .../sdk/options/PipelineOptionsFactoryTest.java    |   2 +-
 .../sdk/options/ProxyInvocationHandlerTest.java    |   2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java       |   2 +-
 .../beam/sdk/runners/TransformHierarchyTest.java   |   2 +-
 .../apache/beam/sdk/runners/TransformTreeTest.java |   2 +-
 .../beam/sdk/schemas/AutoValueSchemaTest.java      |   4 +-
 .../beam/sdk/schemas/JavaBeanSchemaTest.java       |  38 +-
 .../beam/sdk/schemas/JavaFieldSchemaTest.java      |   6 +-
 .../apache/beam/sdk/schemas/SchemaCoderTest.java   |   2 +-
 .../sdk/schemas/transforms/CastValidatorTest.java  |   2 +-
 .../beam/sdk/schemas/transforms/CoGroupTest.java   |   2 +-
 .../beam/sdk/schemas/transforms/ConvertTest.java   |  15 +
 .../beam/sdk/schemas/transforms/GroupTest.java     |   2 +-
 .../beam/sdk/schemas/utils/JavaBeanUtilsTest.java  |  26 +-
 .../beam/sdk/schemas/utils/POJOUtilsTest.java      |   6 +-
 .../beam/sdk/schemas/utils/SchemaZipFoldTest.java  |   2 +-
 .../beam/sdk/schemas/utils/TestJavaBeans.java      |  17 +
 .../apache/beam/sdk/state/StateContextsTest.java   |   2 +-
 .../beam/sdk/testing/CoderPropertiesTest.java      |   2 +-
 .../beam/sdk/testing/CombineFnTesterTest.java      |   2 +-
 .../org/apache/beam/sdk/testing/PAssertTest.java   |   2 +-
 .../beam/sdk/testing/PaneExtractorsTest.java       |   2 +-
 .../beam/sdk/testing/SerializableMatchersTest.java |   2 +-
 .../apache/beam/sdk/testing/StaticWindowsTest.java |   2 +-
 .../apache/beam/sdk/testing/TestPipelineTest.java  |   2 +-
 .../beam/sdk/testing/WindowSupplierTest.java       |   2 +-
 .../apache/beam/sdk/transforms/CombineFnsTest.java |   2 +-
 .../apache/beam/sdk/transforms/CombineTest.java    |   2 +-
 .../org/apache/beam/sdk/transforms/CreateTest.java |   2 +-
 .../apache/beam/sdk/transforms/DistinctTest.java   |   2 +-
 .../org/apache/beam/sdk/transforms/DoFnTest.java   |   2 +-
 .../apache/beam/sdk/transforms/DoFnTesterTest.java |   2 +-
 .../apache/beam/sdk/transforms/FlattenTest.java    |   2 +-
 .../apache/beam/sdk/transforms/GroupByKeyTest.java |   2 +-
 .../apache/beam/sdk/transforms/PTransformTest.java |   2 +-
 .../org/apache/beam/sdk/transforms/ParDoTest.java  |   2 +-
 .../org/apache/beam/sdk/transforms/ReifyTest.java  |   2 +-
 .../beam/sdk/transforms/ReifyTimestampsTest.java   |   2 +-
 .../apache/beam/sdk/transforms/ReshuffleTest.java  |   2 +-
 .../beam/sdk/transforms/SimpleFunctionTest.java    |   2 +-
 .../beam/sdk/transforms/WithFailuresTest.java      |   2 +-
 .../display/DisplayDataEvaluatorTest.java          |   2 +-
 .../display/DisplayDataMatchersTest.java           |   2 +-
 .../sdk/transforms/display/DisplayDataTest.java    |   2 +-
 .../beam/sdk/transforms/join/CoGbkResultTest.java  |   2 +-
 .../beam/sdk/transforms/join/CoGroupByKeyTest.java |   2 +-
 .../beam/sdk/transforms/join/UnionCoderTest.java   |   2 +-
 .../sdk/transforms/reflect/DoFnInvokersTest.java   |   2 +-
 .../sdk/transforms/reflect/DoFnSignaturesTest.java |   2 +-
 .../transforms/reflect/OnTimerInvokersTest.java    |   2 +-
 .../sdk/transforms/windowing/FixedWindowsTest.java |   2 +-
 .../transforms/windowing/IntervalWindowTest.java   |   2 +-
 .../sdk/transforms/windowing/RepeatedlyTest.java   |   2 +-
 .../sdk/transforms/windowing/SessionsTest.java     |   2 +-
 .../transforms/windowing/SlidingWindowsTest.java   |   2 +-
 .../beam/sdk/transforms/windowing/WindowTest.java  |   2 +-
 .../org/apache/beam/sdk/util/ApiSurfaceTest.java   |   2 +-
 .../BufferedElementCountingOutputStreamTest.java   |   2 +-
 .../apache/beam/sdk/util/FluentBackoffTest.java    |   2 +-
 .../org/apache/beam/sdk/util/HistogramTest.java    |   2 +-
 .../org/apache/beam/sdk/util/MoreFuturesTest.java  |   2 +-
 .../org/apache/beam/sdk/util/ReleaseInfoTest.java  |   2 +-
 .../beam/sdk/util/UserCodeExceptionTest.java       |   2 +-
 .../java/org/apache/beam/sdk/util/VarIntTest.java  |   2 +-
 .../apache/beam/sdk/util/WindowedValueTest.java    |   2 +-
 .../org/apache/beam/sdk/util/ZipFilesTest.java     |   2 +-
 .../beam/sdk/util/common/ReflectHelpersTest.java   |   2 +-
 .../java/org/apache/beam/sdk/values/KVTest.java    |   2 +-
 .../beam/sdk/values/PCollectionListTest.java       |   2 +-
 .../beam/sdk/values/PCollectionTupleTest.java      |   2 +-
 .../org/apache/beam/sdk/values/TupleTagTest.java   |   2 +-
 .../beam/sdk/values/TypeDescriptorsTest.java       |   2 +-
 .../apache/beam/sdk/values/TypedPValueTest.java    |   2 +-
 .../beam/sdk/expansion/service/ExternalTest.java   |   2 +-
 .../sdk/extensions/gcp/util/GceMetadataUtil.java   |   4 +-
 .../gcp/storage/GcsFileSystemRegistrarTest.java    |   2 +-
 .../extensions/gcp/storage/GcsFileSystemTest.java  |   2 +-
 .../beam/sdk/extensions/gcp/util/GcsUtilTest.java  |   2 +-
 ...LatencyRecordingHttpRequestInitializerTest.java |   2 +-
 .../gcp/util/RetryHttpRequestInitializerTest.java  |   2 +-
 .../sdk/extensions/gcp/util/gcsfs/GcsPathTest.java |   2 +-
 .../extensions/protobuf/ProtoMessageSchema.java    |  68 +-
 .../extensions/protobuf/ByteStringCoderTest.java   |   2 +-
 .../sdk/extensions/protobuf/ProtobufUtilTest.java  |   2 +-
 .../ExternalSchemaIOTransformRegistrarTest.java    |   4 +-
 .../extensions/sketching/TDigestQuantilesTest.java |   2 +-
 .../extensions/sorter/ExternalSorterBenchmark.java |   6 +-
 .../beam/sdk/extensions/sorter/SortValuesTest.java |   2 +-
 sdks/java/extensions/sql/build.gradle              |  15 +-
 .../datacatalog/DataCatalogTableProvider.java      |  61 +-
 .../sql/meta/provider/datacatalog/SchemaUtils.java |  93 +++
 .../meta/provider/datacatalog/SchemaUtilsTest.java | 157 ++++
 .../sdk/extensions/sql/jdbc/BeamSqlLineIT.java     |   2 +-
 .../sdk/extensions/sql/jdbc/BeamSqlLineTest.java   |   2 +-
 .../sdk/extensions/sql/impl/ScalarFnReflector.java |  73 ++
 .../extensions/sql/impl/ScalarFunctionImpl.java    |  47 +-
 .../sql/meta/provider/bigtable/BigtableFilter.java | 136 ++++
 .../sql/meta/provider/bigtable/BigtableTable.java  |  47 +-
 .../meta/provider/kafka/BeamKafkaThriftTable.java  | 138 ++++
 .../meta/provider/kafka/KafkaTableProvider.java    |  56 +-
 .../sql/BeamSqlDslSqlStdOperatorsTest.java         |   2 +-
 .../sdk/extensions/sql/impl/JdbcDriverTest.java    |   2 +-
 .../extensions/sql/impl/ScalarFnReflectorTest.java | 141 ++++
 .../provider/bigtable/BigtableClientWrapper.java   | 115 +++
 .../meta/provider/bigtable/BigtableFilterTest.java | 114 +++
 .../BigtableTableCreationFailuresTest.java         |   2 +-
 .../provider/bigtable/BigtableTableFlatTest.java   | 132 +--
 .../meta/provider/bigtable/BigtableTableIT.java    | 200 +++++
 .../meta/provider/bigtable/BigtableTableTest.java  |  94 ---
 .../provider/bigtable/BigtableTableTestUtils.java  | 237 ++++++
 .../bigtable/BigtableTableWithRowsTest.java        | 120 ++-
 .../provider/kafka/BeamKafkaTableProtoTest.java    |   2 +-
 ...rotoTest.java => BeamKafkaTableThriftTest.java} |  53 +-
 .../meta/provider/kafka/KafkaTableProviderIT.java  | 214 ++++-
 .../provider/kafka/KafkaTableProviderProtoIT.java  |  53 --
 .../provider/kafka/KafkaTableProviderTest.java     |  55 +-
 .../provider/kafka/thrift/ItThriftMessage.java     | 612 ++++++++++++++
 .../provider/kafka/thrift/SimpleThriftMessage.java | 509 ++++++++++++
 .../provider/kafka/thrift/TestThriftMessage.java   | 881 +++++++++++++++++++++
 .../sql/meta/provider/pubsub/PubsubAvroIT.java     | 102 ---
 .../sql/meta/provider/pubsub/PubsubJsonIT.java     |  76 --
 .../provider/pubsub/PubsubTableProviderIT.java     | 273 +++++--
 .../meta/provider/text/TextTableProviderTest.java  |   2 +-
 .../sql/meta/store/InMemoryMetaStoreTest.java      |   2 +-
 .../sql/src/test/thrift/kafka/messages.thrift      |  36 +-
 sdks/{go => java/extensions/sql/udf}/build.gradle  |  27 +-
 .../beam/sdk/extensions/sql/udf/AggregateFn.java   |  71 ++
 .../beam/sdk/extensions/sql/udf/ScalarFn.java      |  53 ++
 .../beam/sdk/extensions/sql/udf/UdfProvider.java}  |  23 +-
 .../beam/sdk/extensions/sql/udf/package-info.java  |  16 +-
 .../sdk/extensions/sql/zetasql/SqlAnalyzer.java    |   6 +
 .../extensions/sql/zetasql/ZetaSQLPlannerImpl.java |   5 +-
 .../sql/zetasql/translation/ConversionContext.java |   2 +-
 .../zetasql/translation/ExpressionConverter.java   |  11 +-
 .../sql/zetasql/translation/SqlOperators.java      |  76 +-
 .../translation/UserFunctionDefinitions.java       |  30 +-
 .../translation/ZetaSqlScalarFunctionImpl.java     |  86 ++
 .../zetasketch/ApproximateCountDistinct.java       | 288 +++++++
 .../zetasketch/ApproximateCountDistinctTest.java   | 342 ++++++++
 .../sdk/fn/data/BeamFnDataInboundObserver.java     |  35 +-
 .../beam/sdk/fn/data/DecodingFnDataReceiver.java   |  46 ++
 .../org/apache/beam/sdk/fn/IdGeneratorsTest.java   |   2 +-
 .../sdk/fn/channel/SocketAddressFactoryTest.java   |   2 +-
 .../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java |   2 +-
 ...DataSizeBasedBufferingOutboundObserverTest.java |   2 +-
 .../CompletableFutureInboundDataClientTest.java    |   2 +-
 .../beam/sdk/fn/data/RemoteGrpcPortReadTest.java   |   2 +-
 .../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java  |   2 +-
 .../fn/splittabledofn/RestrictionTrackersTest.java |   2 +-
 .../beam/sdk/fn/stream/AdvancingPhaserTest.java    |   2 +-
 .../apache/beam/sdk/fn/stream/DataStreamsTest.java |   2 +-
 .../sdk/fn/stream/OutboundObserverFactoryTest.java |   2 +-
 .../apache/beam/sdk/fn/test/TestStreamsTest.java   |   2 +-
 .../fn/harness/control/ProcessBundleHandler.java   |  87 +-
 .../beam/fn/harness/data/BeamFnDataClient.java     |  14 +-
 .../beam/fn/harness/data/BeamFnDataGrpcClient.java |  10 +-
 .../fn/harness/data/QueueingBeamFnDataClient.java  |  11 +-
 .../fn/harness/logging/BeamFnLoggingClient.java    |  15 +
 .../beam/fn/harness/AssignWindowsRunnerTest.java   |   2 +-
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  |   2 +-
 .../beam/fn/harness/BeamFnDataWriteRunnerTest.java |   2 +-
 .../apache/beam/fn/harness/CombineRunnersTest.java |   2 +-
 .../apache/beam/fn/harness/FlattenRunnerTest.java  |   2 +-
 .../beam/fn/harness/FnApiDoFnRunnerTest.java       |   2 +-
 .../apache/beam/fn/harness/MapFnRunnersTest.java   |   2 +-
 .../fn/harness/PrecombineGroupingTableTest.java    |   2 +-
 .../beam/fn/harness/WindowMergingFnRunnerTest.java |   2 +-
 .../harness/control/BeamFnControlClientTest.java   |   2 +-
 .../harness/control/BundleSplitListenerTest.java   |   2 +-
 .../harness/control/ProcessBundleHandlerTest.java  |   2 +-
 .../fn/harness/data/BeamFnDataGrpcClientTest.java  |   2 +-
 .../data/BeamFnDataInboundObserverTest.java        |  13 +-
 .../data/PCollectionConsumerRegistryTest.java      |   2 +-
 .../harness/data/QueueingBeamFnDataClientTest.java |   2 +-
 .../harness/logging/BeamFnLoggingClientTest.java   |   2 +-
 .../state/BeamFnStateGrpcClientCacheTest.java      |   2 +-
 .../stream/HarnessStreamObserverFactoriesTest.java |   2 +-
 .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java       |  40 +-
 .../beam/sdk/io/aws/coders/AwsCodersTest.java      |   2 +-
 .../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java   |  68 +-
 .../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java  |   2 +-
 .../beam/sdk/io/aws/options/AwsModuleTest.java     |   2 +-
 .../beam/sdk/io/aws/s3/S3FileSystemTest.java       |   2 +-
 .../sdk/io/aws/sns/PublishResultCodersTest.java    |   2 +-
 .../aws2/dynamodb/BasicDynamoDbClientProvider.java |  32 +-
 .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java      |  40 +-
 .../sdk/io/aws2/options/AwsSerializableUtils.java  |  50 ++
 .../io/aws2/sns/BasicSnsAsyncClientProvider.java   |  32 +-
 .../sdk/io/aws2/sns/BasicSnsClientProvider.java    |  34 +-
 .../sdk/io/aws2/sqs/BasicSqsClientProvider.java    |  34 +-
 .../dynamodb/BasicDynamoDbClientProviderTest.java  |  51 ++
 .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java  |  85 +-
 .../beam/sdk/io/aws2/options/AwsModuleTest.java    |   2 +-
 .../io/aws2/options/AwsSerializableUtilsTest.java  |  70 ++
 .../aws2/sns/BasicSnsAsyncClientProviderTest.java  |  51 ++
 .../io/aws2/sns/BasicSnsClientProviderTest.java    |  51 ++
 .../io/aws2/sqs/BasicSqsClientProviderTest.java    |  51 ++
 .../blobstore/AzureBlobStoreFileSystemTest.java    |   2 +-
 .../beam/sdk/io/cassandra/CassandraIOTest.java     |   2 +-
 .../io/contextualtextio/ContextualTextIOTest.java  |   2 +-
 .../elasticsearch/ElasticsearchIOTestCommon.java   |   2 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java       |   7 +-
 .../gcp/bigquery/BigQueryStorageStreamSource.java  |   6 +
 .../apache/beam/sdk/io/gcp/bigtable/RowUtils.java  |  10 +
 .../apache/beam/sdk/io/gcp/healthcare/FhirIO.java  | 232 ++++++
 .../sdk/io/gcp/healthcare/HealthcareApiClient.java |  17 +
 .../io/gcp/healthcare/HttpHealthcareApiClient.java | 174 ++++
 .../sdk/io/gcp/healthcare/JsonArrayCoder.java}     |  34 +-
 .../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java |  28 +-
 .../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java  |  17 +-
 .../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java   | 150 ++--
 .../sdk/io/gcp/pubsublite/CloudPubsubChecks.java   |  21 +-
 ...fsetFinalizer.java => InitialOffsetReader.java} |  12 +-
 .../io/gcp/pubsublite/InitialOffsetReaderImpl.java |  54 ++
 .../gcp/pubsublite/LimitingTopicBacklogReader.java |  81 ++
 ...ffsetFinalizer.java => OffsetByteProgress.java} |  17 +-
 .../io/gcp/pubsublite/OffsetByteRangeTracker.java  | 172 ++++
 .../io/gcp/pubsublite/OffsetCheckpointMark.java    |  78 --
 .../pubsublite/PerSubscriptionPartitionSdf.java    | 110 +++
 .../beam/sdk/io/gcp/pubsublite/PublisherCache.java |  10 +-
 .../sdk/io/gcp/pubsublite/PublisherOptions.java    |  47 +-
 .../sdk/io/gcp/pubsublite/PublisherOrError.java    |   6 +-
 .../beam/sdk/io/gcp/pubsublite/Publishers.java     |  56 ++
 .../beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java   |  14 +-
 .../beam/sdk/io/gcp/pubsublite/PubsubLiteSink.java |  18 +-
 .../gcp/pubsublite/PubsubLiteUnboundedReader.java  | 333 --------
 .../gcp/pubsublite/PubsubLiteUnboundedSource.java  | 140 ----
 ...zer.java => SerializableSubscriberFactory.java} |  17 +-
 .../io/gcp/pubsublite/SerializableSupplier.java    |   5 +-
 .../sdk/io/gcp/pubsublite/SubscribeTransform.java  | 124 +++
 .../sdk/io/gcp/pubsublite/SubscriberOptions.java   | 157 ++--
 ...etFinalizer.java => SubscriptionPartition.java} |  19 +-
 ...dCoder.java => SubscriptionPartitionCoder.java} |  36 +-
 ...er.java => SubscriptionPartitionProcessor.java} |  16 +-
 ... => SubscriptionPartitionProcessorFactory.java} |  15 +-
 .../SubscriptionPartitionProcessorImpl.java        | 152 ++++
 .../sdk/io/gcp/pubsublite/TopicBacklogReader.java  |  34 +-
 .../io/gcp/pubsublite/TopicBacklogReaderImpl.java  |  80 +-
 .../gcp/pubsublite/TopicBacklogReaderSettings.java |  50 +-
 .../gcp/pubsublite/TranslatingPullSubscriber.java  |  48 --
 .../beam/sdk/io/gcp/pubsublite/UuidCoder.java      |   7 +-
 .../gcp/pubsublite/UuidDeduplicationOptions.java   |  15 +-
 .../beam/sdk/io/gcp/spanner/SpannerAccessor.java   |  22 +-
 .../io/gcp/testing/BigtableEmulatorWrapper.java    |  75 --
 .../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 154 ----
 .../beam/sdk/io/gcp/testing/BigtableUtils.java     |  52 +-
 .../cloud/spanner/FakeBatchTransactionId.java      |   1 +
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |   3 +-
 .../sdk/io/gcp/bigquery/BigQueryIOReadTest.java    |   2 +-
 .../io/gcp/bigquery/BigQueryIOStorageReadTest.java |  13 +-
 .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java   |   2 +-
 .../io/gcp/bigquery/BigQueryServicesImplTest.java  |   2 +-
 .../sdk/io/gcp/bigquery/BigQueryUtilsTest.java     |   2 +-
 .../sdk/io/gcp/bigquery/TableRowJsonCoderTest.java |   2 +-
 .../bigtable/BeamRowToBigtableMutationTest.java    |  14 +-
 .../sdk/io/gcp/bigtable/BigtableConfigTest.java    |   2 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   |   2 +-
 .../gcp/bigtable/BigtableRowToBeamRowFlatTest.java |  14 +-
 .../io/gcp/bigtable/BigtableRowToBeamRowTest.java  |  17 +-
 .../{TestUtils.java => BigtableTestUtils.java}     |  50 +-
 .../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java  |   2 +-
 .../sdk/io/gcp/bigtable/CellValueParserTest.java   |  13 +-
 .../io/gcp/datastore/AdaptiveThrottlerTest.java    |   2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java |   2 +-
 .../gcp/datastore/EntityToRowRowToEntityTest.java  |   6 +-
 .../beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 156 ++++
 .../beam/sdk/io/gcp/healthcare/FhirIOTest.java     |  21 +
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java       |   2 +-
 .../pubsub/PubsubMessagePayloadOnlyCoderTest.java  |   2 +-
 ...MessageWithAttributesAndMessageIdCoderTest.java |   2 +-
 .../PubsubMessageWithAttributesCoderTest.java      |   2 +-
 .../PubsubMessageWithMessageIdCoderTest.java       |   2 +-
 .../io/gcp/pubsub/PubsubUnboundedSourceTest.java   |   2 +-
 .../gcp/pubsublite/OffsetByteRangeTrackerTest.java | 158 ++++
 .../gcp/pubsublite/OffsetCheckpointMarkTest.java   |  81 --
 .../PerSubscriptionPartitionSdfTest.java           | 189 +++++
 .../sdk/io/gcp/pubsublite/PubsubLiteSinkTest.java  |  47 +-
 .../pubsublite/PubsubLiteUnboundedReaderTest.java  | 330 --------
 .../SubscriptionPartitionProcessorImplTest.java    | 227 ++++++
 .../gcp/pubsublite/TopicBacklogReaderImplTest.java | 140 +---
 .../pubsublite/UuidDeduplicationTransformTest.java |  13 +-
 .../io/gcp/spanner/MutationSizeEstimatorTest.java  |   2 +-
 .../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java  |   2 +-
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java     |   2 +-
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java    |   2 +-
 .../beam/sdk/io/gcp/spanner/StructUtilsTest.java   |   2 +-
 .../sdk/io/hadoop/SerializableConfiguration.java   |  11 +
 .../io/hadoop/SerializableConfigurationTest.java   |   9 +
 .../beam/sdk/io/hadoop/WritableCoderTest.java      |   2 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemOptions.java  |   9 +-
 .../sdk/io/hdfs/HadoopFileSystemModuleTest.java    |   2 +-
 .../hdfs/HadoopFileSystemOptionsRegistrarTest.java |   2 +-
 .../sdk/io/hdfs/HadoopFileSystemOptionsTest.java   |  22 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemTest.java     |   2 +-
 .../beam/sdk/io/hadoop/format/HadoopFormatIO.java  |  85 +-
 .../hadoop/format/HadoopFormatIOCassandraTest.java |   6 +-
 .../io/hadoop/format/HadoopFormatIOReadTest.java   | 162 +++-
 .../org/apache/beam/sdk/io/hbase/HBaseIOTest.java  |   2 +-
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java       |   2 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java    |   2 +-
 .../java/org/apache/beam/sdk/io/jms/JmsIOTest.java |   2 +-
 .../CustomTimestampPolicyWithLimitedDelayTest.java |   2 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  |   8 +-
 .../apache/beam/sdk/io/kinesis/KinesisIOIT.java    |   2 +-
 .../java/org/apache/beam/sdk/io/kudu/KuduIOIT.java |   4 +-
 .../sdk/io/parquet/GenericRecordReadConverter.java |  64 --
 .../org/apache/beam/sdk/io/parquet/ParquetIO.java  | 472 +++++++++--
 .../sdk/io/parquet/ParquetSchemaIOProvider.java    |  11 +-
 .../sdk/io/parquet/GenericRecordToRowTest.java     |  79 --
 .../apache/beam/sdk/io/parquet/ParquetIOTest.java  | 132 ++-
 .../apache/beam/sdk/io/snowflake/KeyPairUtils.java |   4 +-
 .../io/snowflake/crosslanguage/ReadBuilder.java    |   4 +-
 .../services/SnowflakeBatchServiceImpl.java        |   4 +-
 .../test/FakeSnowflakeStreamingServiceImpl.java    |   4 +-
 .../beam/sdk/io/snowflake/test/TestUtils.java      |   6 +-
 .../beam/sdk/io/solr/JavaBinCodecCoderTest.java    |   2 +-
 .../org/apache/beam/sdk/io/solr/SolrIOTest.java    |   4 +-
 sdks/java/io/thrift/build.gradle                   |  10 +-
 .../org/apache/beam/sdk/io/thrift/ThriftCoder.java |   9 +-
 .../apache/beam/sdk/io/thrift/ThriftSchema.java    | 408 ++++++++++
 .../apache/beam/sdk/io/thrift/TestThriftEnum.java} |  45 +-
 .../beam/sdk/io/thrift/TestThriftInnerStruct.java  | 526 ++++++++++++
 .../beam/sdk/io/thrift/TestThriftStruct.java       | 760 ++++++++++++++++--
 .../apache/beam/sdk/io/thrift/TestThriftUnion.java | 401 ++++++++++
 .../apache/beam/sdk/io/thrift/ThriftIOTest.java    |  38 +-
 .../beam/sdk/io/thrift/ThriftSchemaTest.java       | 251 ++++++
 .../src/test/resources/thrift/thrift_test.thrift   |  35 +-
 .../apache/beam/sdk/io/tika/ParseResultTest.java   |   2 +-
 .../org/apache/beam/sdk/io/tika/TikaIOTest.java    |   2 +-
 .../java/org/apache/beam/sdk/io/xml/XmlSource.java |  22 +-
 .../org/apache/beam/sdk/io/xml/JAXBCoderTest.java  |   2 +-
 .../java/org/apache/beam/sdk/io/xml/XmlIOTest.java |  11 +-
 .../org/apache/beam/sdk/io/xml/XmlSourceTest.java  |  44 +-
 sdks/java/maven-archetypes/examples/build.gradle   |   2 +-
 sdks/java/testing/jpms-tests/build.gradle          |   2 +-
 .../java/org/apache/beam/sdk/nexmark/Monitor.java  |  13 +-
 .../apache/beam/sdk/nexmark/NexmarkLauncher.java   |  50 +-
 .../apache/beam/sdk/nexmark/NexmarkOptions.java    |  14 +
 .../beam/sdk/nexmark/queries/NexmarkQuery.java     |   3 -
 .../sdk/nexmark/queries/NexmarkQueryModel.java     |   4 +-
 .../apache/beam/sdk/nexmark/queries/Query10.java   |   8 +-
 .../apache/beam/sdk/nexmark/queries/Query3.java    | 191 +++--
 .../apache/beam/sdk/nexmark/queries/Query5.java    | 133 +++-
 .../beam/sdk/nexmark/NexmarkConfigurationTest.java |   2 +-
 .../beam/sdk/nexmark/PerfsToBigQueryTest.java      |   2 +-
 .../nexmark/queries/BoundedSideInputJoinTest.java  |   2 +-
 .../nexmark/queries/SessionSideInputJoinTest.java  |   2 +-
 .../queries/sql/SqlBoundedSideInputJoinTest.java   |   2 +-
 sdks/python/apache_beam/coders/coders.py           |  71 +-
 .../apache_beam/coders/coders_test_common.py       |   1 -
 sdks/python/apache_beam/coders/typecoders.py       |   3 -
 sdks/python/apache_beam/dataframe/frames.py        |  26 +-
 sdks/python/apache_beam/dataframe/frames_test.py   |  78 +-
 sdks/python/apache_beam/dataframe/io.py            | 263 ++++--
 sdks/python/apache_beam/dataframe/io_test.py       |  88 +-
 .../examples/cookbook/bigquery_side_input.py       |   7 +-
 .../apache_beam/examples/cookbook/filters.py       |   2 +-
 .../apache_beam/examples/snippets/snippets.py      |  16 +-
 .../apache_beam/examples/snippets/snippets_test.py |  10 +-
 sdks/python/apache_beam/examples/sql_taxi.py       |   3 +-
 sdks/python/apache_beam/io/concat_source.py        |   9 +-
 sdks/python/apache_beam/io/concat_source_test.py   |   8 +
 .../flink/flink_streaming_impulse_source_test.py   |   3 +-
 sdks/python/apache_beam/io/gcp/bigquery.py         |   2 +-
 .../apache_beam/io/gcp/bigquery_file_loads.py      |  21 +-
 .../apache_beam/io/gcp/bigquery_read_internal.py   |  40 +-
 sdks/python/apache_beam/io/gcp/bigquery_test.py    |  14 +-
 sdks/python/apache_beam/io/gcp/pubsub.py           |   8 +-
 sdks/python/apache_beam/io/gcp/pubsub_test.py      |   3 +-
 sdks/python/apache_beam/io/mongodbio.py            |  29 +-
 sdks/python/apache_beam/io/mongodbio_test.py       |  41 +
 sdks/python/apache_beam/io/restriction_trackers.py |   2 +-
 sdks/python/apache_beam/metrics/execution.pxd      |   1 +
 sdks/python/apache_beam/metrics/execution.py       |   9 +-
 .../python/apache_beam/options/pipeline_options.py |  29 +-
 sdks/python/apache_beam/pipeline.py                | 163 ++--
 sdks/python/apache_beam/portability/common_urns.py |   1 +
 sdks/python/apache_beam/runners/common.pxd         |   1 +
 sdks/python/apache_beam/runners/common.py          |  36 +-
 .../runners/dataflow/dataflow_runner.py            | 184 ++---
 .../runners/dataflow/dataflow_runner_test.py       |  62 ++
 .../apache_beam/runners/dataflow/internal/names.py |   9 +-
 .../runners/dataflow/ptransform_overrides.py       |  45 ++
 .../apache_beam/runners/direct/direct_runner.py    |   4 +-
 .../apache_beam/runners/direct/direct_userstate.py |  35 +-
 .../runners/direct/transform_evaluator.py          |   8 +-
 sdks/python/apache_beam/runners/direct/util.py     |  18 +-
 .../runners/direct/watermark_manager.py            |  10 +-
 .../python/apache_beam/runners/pipeline_context.py |   2 -
 .../runners/portability/flink_runner_test.py       |   3 +
 .../portability/fn_api_runner/fn_runner_test.py    |  24 +
 .../portability/fn_api_runner/translations.py      |  81 +-
 .../portability/fn_api_runner/translations_test.py |  18 +-
 .../portability/fn_api_runner/worker_handlers.py   |   7 +-
 .../apache_beam/runners/portability/job_server.py  |  10 +-
 .../runners/portability/job_server_test.py         |  84 ++
 .../runners/portability/portable_runner.py         |  85 +-
 .../runners/portability/sdk_container_builder.py   |  11 +-
 .../runners/portability/spark_runner_test.py       |   5 +-
 sdks/python/apache_beam/runners/runner.py          |  24 +-
 .../apache_beam/runners/worker/bundle_processor.py |  41 +-
 .../apache_beam/runners/worker/operations.py       |   6 +-
 sdks/python/apache_beam/transforms/core.py         | 145 +---
 sdks/python/apache_beam/transforms/environments.py |  26 +-
 sdks/python/apache_beam/transforms/external.py     |  24 +-
 sdks/python/apache_beam/transforms/ptransform.py   |  46 +-
 sdks/python/apache_beam/transforms/trigger.py      |  39 +-
 sdks/python/apache_beam/transforms/trigger_test.py |  10 +-
 sdks/python/apache_beam/transforms/userstate.py    |  55 +-
 .../apache_beam/transforms/userstate_test.py       | 216 ++++-
 sdks/python/apache_beam/transforms/util.py         | 123 ++-
 sdks/python/apache_beam/transforms/util_test.py    |  65 ++
 .../typehints/native_type_compatibility.py         |   7 -
 .../apache_beam/typehints/sharded_key_type.py      |  31 +-
 .../apache_beam/typehints/sharded_key_type_test.py |  35 +-
 sdks/python/apache_beam/version.py                 |   2 +-
 sdks/python/container/common.gradle                |   2 +
 sdks/python/setup.py                               |   6 +-
 sdks/python/test-suites/dataflow/build.gradle      |  24 +
 sdks/python/test-suites/gradle.properties          |   5 +
 sdks/python/test-suites/portable/common.gradle     |  34 +-
 settings.gradle                                    |   5 +
 start-build-env.sh                                 |  16 +-
 website/.gitignore                                 |  23 -
 website/www/package.json                           |   2 +-
 website/www/site/config.toml                       |   2 +-
 website/www/site/content/en/blog/beam-2.13.0.md    |   2 +-
 website/www/site/content/en/blog/beam-2.21.0.md    |   2 +-
 website/www/site/content/en/blog/beam-2.22.0.md    |   2 +-
 website/www/site/content/en/blog/beam-2.23.0.md    |   4 +-
 website/www/site/content/en/blog/beam-2.24.0.md    |   4 +-
 website/www/site/content/en/blog/beam-2.25.0.md    |   4 +-
 website/www/site/content/en/blog/beam-2.26.0.md    |  65 ++
 website/www/site/content/en/blog/beam-2.27.0.md    |  69 ++
 .../en/blog/dataframe-api-preview-available.md     | 178 +++++
 .../site/content/en/blog/pattern-match-beam-sql.md |   2 +-
 .../en/blog/splittable-do-fn-is-available.md       |  91 +++
 website/www/site/content/en/contribute/_index.md   |   2 +-
 .../site/content/en/contribute/release-guide.md    |   7 +-
 .../en/documentation/dsls/dataframes/overview.md   | 131 +++
 .../dsls/sql/extensions/create-external-table.md   |  15 +-
 .../content/en/documentation/dsls/sql/overview.md  |   2 +-
 .../documentation/io/built-in/google-bigquery.md   |  15 +-
 .../en/documentation/io/built-in/snowflake.md      |   6 +-
 .../content/en/documentation/programming-guide.md  |   4 +-
 .../content/en/documentation/runners/direct.md     | 112 +--
 .../site/content/en/documentation/runners/flink.md |  42 +-
 .../site/content/en/documentation/runners/samza.md |   5 +
 .../en/documentation/runtime/environments.md       | 305 ++++---
 .../site/content/en/get-started/beam-overview.md   |   5 +-
 .../www/site/content/en/get-started/downloads.md   |  28 +-
 .../www/site/content/en/get-started/from-spark.md  | 268 +++++++
 website/www/site/data/authors.yml                  |  38 +-
 website/www/site/data/capability_matrix.yaml       | 714 +++++++++++++++--
 .../www/site/data/capability_matrix_snapshot.yaml  | 195 +++++
 website/www/site/layouts/blog/single.html          |   1 +
 .../partials/section-menu/en/get-started.html      |   7 +-
 .../layouts/partials/section-menu/en/sdks.html     |  17 +-
 website/www/site/static/.htaccess                  |   2 +-
 941 files changed, 26256 insertions(+), 6803 deletions(-)
 copy .test-infra/jenkins/{job_LoadTests_Combine_Flink_Go.groovy => job_LoadTests_Combine_Go.groovy} (56%)
 create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Go.groovy
 create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Go.groovy
 create mode 100644 .test-infra/jenkins/job_LoadTests_SideInput_Flink_Go.groovy
 create mode 100644 .test-infra/jenkins/job_LoadTests_SideInput_Go.groovy
 create mode 100644 .test-infra/jenkins/job_LoadTests_coGBK_Go.groovy
 rename .test-infra/jenkins/{job_PostCommit_Python36.groovy => job_PostCommit_Python.groovy} (51%)
 delete mode 100644 .test-infra/jenkins/job_PostCommit_Python37.groovy
 delete mode 100644 .test-infra/jenkins/job_PostCommit_Python38.groovy
 copy runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OutputManagerFactory.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaPubsubConstants.java (64%)
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsub.java
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/README.md
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/AvroDataClass.java
 rename sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderJsonIT.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/AvroDataClassKafkaAvroDeserializer.java (53%)
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/package-info.java (74%)
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/SslConsumerFactoryFn.java
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/package-info.java (74%)
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options/KafkaToPubsubOptions.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableStatusFunction.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options/package-info.java (72%)
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/package-info.java (74%)
 create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/transforms/FormatTransform.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/transforms/package-info.java (74%)
 create mode 100644 examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubTest.java
 copy {sdks/go => runners/flink/1.12}/build.gradle (50%)
 copy runners/flink/{1.9 => 1.12}/job-server-container/build.gradle (100%)
 copy {sdks/go => runners/flink/1.12/job-server}/build.gradle (53%)
 create mode 100644 runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
 copy runners/flink/{1.10 => 1.12}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java (80%)
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobInvocation.java
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamJobCoordinatorRunner.java
 rename runners/samza/src/main/java/org/apache/beam/runners/samza/container/{ContainerCfgFactory.java => ContainerCfgLoader.java} (79%)
 copy runners/samza/src/main/java/org/apache/beam/runners/samza/{runtime/OutputManagerFactory.java => container/ContainerCfgLoaderFactory.java} (65%)
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/BundleManager.java
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/FutureCollector.java
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamSystemFactory.java
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamTranslator.java
 create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/FutureUtils.java
 create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidatorTest.java
 create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/BundleManagerTest.java
 create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/FutureCollectorImplTest.java
 create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/translation/TranslationContextTest.java
 create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/FutureUtilsTest.java
 create mode 100644 sdks/go/pkg/beam/core/graph/coder/double_test.go
 create mode 100644 sdks/go/pkg/beam/core/graph/coder/row_decoder.go
 create mode 100644 sdks/go/pkg/beam/core/graph/coder/row_encoder.go
 create mode 100644 sdks/go/pkg/beam/core/graph/coder/testutil/testutil.go
 create mode 100644 sdks/go/pkg/beam/core/graph/coder/testutil/testutil_test.go
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics.go
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics_test.go
 rename sdks/go/test/integration/{ => driver}/driver.go (100%)
 create mode 100644 sdks/go/test/integration/integration.go
 create mode 100644 sdks/go/test/integration/synthetic/synthetic_test.go
 create mode 100644 sdks/go/test/integration/xlang/xlang_test.go
 create mode 100644 sdks/go/test/load/sideinput/sideinput.go
 copy sdks/go/test/{run_integration_tests.sh => run_validatesrunner_tests.sh} (57%)
 delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenericRecordWriteConverter.java
 create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/RowMessages.java
 create mode 100644 sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtilsTest.java
 create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflector.java
 create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilter.java
 create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaThriftTable.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflectorTest.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableClientWrapper.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilterTest.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableIT.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableTest.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableTestUtils.java
 copy sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/{BeamKafkaTableProtoTest.java => BeamKafkaTableThriftTest.java} (63%)
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderProtoIT.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/ItThriftMessage.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/SimpleThriftMessage.java
 create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/TestThriftMessage.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubAvroIT.java
 delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
 copy runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineOptions.java => sdks/java/extensions/sql/src/test/thrift/kafka/messages.thrift (54%)
 copy sdks/{go => java/extensions/sql/udf}/build.gradle (56%)
 create mode 100644 sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java
 create mode 100644 sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/ScalarFn.java
 copy sdks/java/{io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetFinalizer.java => extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/UdfProvider.java} (57%)
 copy runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OutputManagerFactory.java => sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/package-info.java (68%)
 create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
 create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/ApproximateCountDistinct.java
 create mode 100644 sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/ApproximateCountDistinctTest.java
 create mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/DecodingFnDataReceiver.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsSerializableUtils.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProviderTest.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsSerializableUtilsTest.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsAsyncClientProviderTest.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProviderTest.java
 create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/BasicSqsClientProviderTest.java
 rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderCSVIT.java => io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/JsonArrayCoder.java} (52%)
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => InitialOffsetReader.java} (77%)
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/InitialOffsetReaderImpl.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/LimitingTopicBacklogReader.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => OffsetByteProgress.java} (68%)
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetByteRangeTracker.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetCheckpointMark.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PerSubscriptionPartitionSdf.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/Publishers.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedReader.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedSource.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SerializableSubscriberFactory.java} (64%)
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscribeTransform.java
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SubscriptionPartition.java} (64%)
 copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{UuidCoder.java => SubscriptionPartitionCoder.java} (50%)
 rename sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SubscriptionPartitionProcessor.java} (69%)
 rename sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{SerializableStatusFunction.java => SubscriptionPartitionProcessorFactory.java} (60%)
 create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscriptionPartitionProcessorImpl.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/TranslatingPullSubscriber.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableEmulatorWrapper.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableTestUtils.java
 copy runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java => sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableUtils.java (54%)
 rename sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/{TestUtils.java => BigtableTestUtils.java} (71%)
 create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetByteRangeTrackerTest.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetCheckpointMarkTest.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/PerSubscriptionPartitionSdfTest.java
 delete mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedReaderTest.java
 create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscriptionPartitionProcessorImplTest.java
 delete mode 100644 sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/GenericRecordReadConverter.java
 delete mode 100644 sdks/java/io/parquet/src/test/java/org/apache/beam/sdk/io/parquet/GenericRecordToRowTest.java
 create mode 100644 sdks/java/io/thrift/src/main/java/org/apache/beam/sdk/io/thrift/ThriftSchema.java
 rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderAvroIT.java => io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftEnum.java} (50%)
 create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftInnerStruct.java
 create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftUnion.java
 create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/ThriftSchemaTest.java
 create mode 100644 sdks/python/apache_beam/runners/portability/job_server_test.py
 delete mode 100644 website/.gitignore
 create mode 100644 website/www/site/content/en/blog/beam-2.26.0.md
 create mode 100644 website/www/site/content/en/blog/beam-2.27.0.md
 create mode 100644 website/www/site/content/en/blog/dataframe-api-preview-available.md
 create mode 100644 website/www/site/content/en/blog/splittable-do-fn-is-available.md
 create mode 100644 website/www/site/content/en/documentation/dsls/dataframes/overview.md
 create mode 100644 website/www/site/content/en/get-started/from-spark.md


Mime
View raw message