beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4835) Add more flexible options for data loading to BigQueryIO.Write
Date Wed, 01 Aug 2018 10:00:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4835?focusedWorklogId=129640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129640
]

ASF GitHub Bot logged work on BEAM-4835:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Aug/18 09:59
            Start Date: 01/Aug/18 09:59
    Worklog Time Spent: 10m 
      Work Description: gene-telligent commented on issue #6008: [BEAM-4835] Adding in additional
options for BigQueryIO insert statements
URL: https://github.com/apache/beam/pull/6008#issuecomment-409521605
 
 
   @lukecwik Rebased and ready to merge.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 129640)
    Time Spent: 0.5h  (was: 20m)

> Add more flexible options for data loading to BigQueryIO.Write
> --------------------------------------------------------------
>
>                 Key: BEAM-4835
>                 URL: https://issues.apache.org/jira/browse/BEAM-4835
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>            Reporter: Gene Peters
>            Assignee: Chamikara Jayalath
>            Priority: Minor
>              Labels: pull-request-available
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> As part of the BigQuery API, there are a few options exposed to end-users which allow
for more flexible data loading.
> For both [streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-]
and [batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-]
inserts, the flag "ignoreUnknownValues" can be set, which indicates if BigQuery should accept
rows that contain values that do not match the schema. 
> [In addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-]
streaming inserts allow for the option of accepting an inserted batch of rows even if some
of of the rows are invalid.
> I've made the necessary code changes to make this available within BigQueryIO.Write and
will be attaching the pull request to this ticket for review. Both flags are off by default.
> Let me know if you have any questions or feedback about this!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message