beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Lambert (JIRA)" <>
Subject [jira] [Created] (BEAM-1800) Can't save datastore objects
Date Fri, 24 Mar 2017 08:01:41 GMT
Mike Lambert created BEAM-1800:

             Summary: Can't save datastore objects
                 Key: BEAM-1800
             Project: Beam
          Issue Type: Bug
          Components: sdk-py
            Reporter: Mike Lambert
            Assignee: Ahmet Altay

I can't seem to save my database objects using {{WriteToDatastore}}, as it errors out on a
strange unicode issue when trying to write a batch. Stacktrace follows:

File "apache_beam/runners/", line 195, in apache_beam.runners.common.DoFnRunner.receive
File "apache_beam/runners/", line 267, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/", line 279, in apache_beam.runners.common.DoFnRunner.reraise_augmented
  raise type(exn), args, sys.exc_info()[2] 
File "apache_beam/runners/", line 263, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/", line 198, in apache_beam.runners.common.DoFnRunner._dofn_simple_invoker
  self._process_outputs(element, self.dofn_process(element.value)) 
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/",
line 354, in process
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/",
line 363, in _flush_batch
  helper.write_mutations(self._datastore, self._project, self._mutations) 
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/", line
187, in write_mutations
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/", line 174, in wrapper
  return fun(*args, **kwargs) 
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/", line
185, in commit
File "/usr/local/lib/python2.7/dist-packages/googledatastore/", line 140, in
File "/usr/local/lib/python2.7/dist-packages/googledatastore/", line 199, in
  method='POST', body=payload, headers=headers) 
File "/usr/local/lib/python2.7/dist-packages/oauth2client/", line 631, in new_request
  redirections, connection_type) 
File "/usr/local/lib/python2.7/dist-packages/httplib2/", line 1609, in request
(response, content)
  = self._request(conn, authority, uri, request_uri, method, body, headers, redirections,
File "/usr/local/lib/python2.7/dist-packages/httplib2/", line 1351, in _request
(response, content)
  = self._conn_request(conn, request_uri, method, body, headers) 
File "/usr/local/lib/python2.7/dist-packages/httplib2/", line 1273, in _conn_request
  conn.request(method, request_uri, body, headers) 
File "/usr/lib/python2.7/", line 1039, in request
  self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/", line 1073, in _send_request
File "/usr/lib/python2.7/", line 1035, in endheaders
File "/usr/lib/python2.7/", line 877, in _send_output
  msg += message_body TypeError: must be str, not unicode
[while running 'write to datastore/Convert to Mutation']

My code is basically:
        | 'convert from entity' >> beam.Map(ConvertFromEntity)
        | 'write to datastore' >> WriteToDatastore(client.project)

Where {{ConvertFromEntity}} converts from a object (which has a nice
API/interface) into the underlying protobuf (which is what the beam gcp/datastore library
from import helpers
def ConvertFromEntity(entity):
    return helpers.entity_to_protobuf(entity)

I assume entity_to_protobuf works fine/normally, since it's also what is used by {google/cloud/datastore/}
to write a bunch of {entity_pb2.Entity} objects into the {datastore_pb2.CommitRequest.mutations[n].upsert}:

In {put() -> _assign_entity_to_pb() -> entity_to_protobuf()}.

In {WriteToDatastore->DatastoreWriteFn.to_upsert_mutation->_Mutate.DatastoreWriteFn->helper.write_mutations}

Any idea what's going on here and why this doesn't work? Yes, I may have some unicode in my
objects...but it works in my appengine DB/NDB usage.

I will attempt to skip WriteToDatastore and just put unbatched entities using the datastore
library and see if that goes any better for me...

This message was sent by Atlassian JIRA

View raw message