beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3401) Fully specify the length prefix coder
Date Tue, 02 Jan 2018 23:35:00 GMT
Luke Cwik created BEAM-3401:
-------------------------------

             Summary: Fully specify the length prefix coder
                 Key: BEAM-3401
                 URL: https://issues.apache.org/jira/browse/BEAM-3401
             Project: Beam
          Issue Type: Improvement
          Components: sdk-go, sdk-java-core, sdk-py-core
            Reporter: Luke Cwik
            Priority: Minor


Choose a variable integer encoding that is efficient in encoding/decoding speed out of one
of the many variants like LEB128 / PrefixVarint / SQLite varint

Also update the length prefix encoding to be parameterized by a block size so that the entire
object doesn't have to be written in memory as per [http://doc/1IGduUqmhWDi_69l9nG8kw73HZ5WI5wOps9Tshl5wpQA/edit#heading=h.j8zpiclux4id]

Add cross language validation tests for encoding/decoding to [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/data/standard_coders.yaml]

Document the format explicitly and choose a stable URN.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message