beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thang Nguyen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3218) Change PubsubBoundedWriter to check total byte size
Date Tue, 21 Nov 2017 16:54:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261035#comment-16261035
] 

Thang Nguyen commented on BEAM-3218:
------------------------------------

Should there be additional checks to assert that an individual {{PubsubMessage}} is adhering
to the [quotas|https://cloud.google.com/pubsub/quotas#other_limits] as well?

Rephrased, if any of the following are true when initializing a {{PubsubMessage}}, should
an exception be thrown?

* more than 100 attributes 
* attr key > 256 bytes
* attr val > 1024 bytes
* single message size > 10mb

This may be out of scope for this ticket, but I just wanted to call it out.

> Change PubsubBoundedWriter to check total byte size
> ---------------------------------------------------
>
>                 Key: BEAM-3218
>                 URL: https://issues.apache.org/jira/browse/BEAM-3218
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Thang Nguyen
>            Assignee: Reuven Lax
>            Priority: Minor
>
> The PubsubBoundedWriter [does not check|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L895-L897]
the total byte size of outgoing messages when publishing. 
> AC:
> * Add a check to ensure the total bytes of the outgoing messages is less than pubsub's
allowed max



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message