beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oscar Korz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3569) SpannerIO.write throws on delete mutations
Date Tue, 30 Jan 2018 18:32:01 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345572#comment-16345572
] 

Oscar Korz commented on BEAM-3569:
----------------------------------

It looks like a fix for this was merged to master already before 2.2.0 was released (2017-12-02).
When can we expect this patch to be in a released version?

{noformat}
commit 3ba96003d31ce98a54c0c51c1c0a9cf7c06e2fa2
Author: Mairbek Khadikov <mairbek@google.com>
Date:   2017-10-18 15:26:11 -0700

    [BEAM-1542] SpannerIO: mutation encoding and size estimation improvements

     static long sizeOf(Mutation m) {
+    if (m.getOperation() == Mutation.Op.DELETE) {
+      return sizeOf(m.getKeySet());
+    }
{noformat}

> SpannerIO.write throws on delete mutations
> ------------------------------------------
>
>                 Key: BEAM-3569
>                 URL: https://issues.apache.org/jira/browse/BEAM-3569
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>    Affects Versions: 2.2.0
>            Reporter: Oscar Korz
>            Assignee: Kenneth Knowles
>            Priority: Major
>         Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. The exception
is generated by trying to guess the size of a delete mutation which cannot contain any values
(deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a DELETE
mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to either
0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API directly in a custom
DoFn, but this forces users to either reimplement all the intelligent batching that SpannerIO
does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message