beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2768) Fix bigquery.WriteTables generating non-unique job identifiers
Date Wed, 16 Aug 2017 05:00:02 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128315#comment-16128315
] 

ASF GitHub Bot commented on BEAM-2768:
--------------------------------------

GitHub user reuvenlax opened a pull request:

    https://github.com/apache/beam/pull/3722

    [BEAM-2768] Ensure that each triggered load generates a different job id 

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/reuvenlax/incubator-beam fix_load_jobs_in_streaming

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3722.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3722
    
----
commit 6d60c2d978ae4042d643bc090ba3b18f0c139e2b
Author: Reuven Lax <relax@relax-macbookpro2.roam.corp.google.com>
Date:   2017-08-16T04:58:14Z

    Ensure that each triggered load generates a different job id (for the case of streaming
triggered file loads), and add test coverage to catch this.

----


> Fix bigquery.WriteTables generating non-unique job identifiers
> --------------------------------------------------------------
>
>                 Key: BEAM-2768
>                 URL: https://issues.apache.org/jira/browse/BEAM-2768
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model
>    Affects Versions: 2.0.0
>            Reporter: Matti Remes
>            Assignee: Reuven Lax
>
> This is a result of BigQueryIO not creating unique job ids for batch inserts, thus BigQuery
API responding with a 409 conflict error:
> {code:java}
> Request failed with code 409, will NOT retry: https://www.googleapis.com/bigquery/v2/projects/<project_id>/jobs
> {code}
> The jobs are initiated in a step BatchLoads/SinglePartitionWriteTables, called by step's
WriteTables ParDo:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L511-L521
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L148
> It would probably be a good idea to append a UUIDs as part of a job id.
> Edit: This is a major bug blocking using BigQuery as a sink for bounded input.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message