beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [beam] pabloem commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro
Date Wed, 11 Mar 2020 22:04:51 GMT
pabloem commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro
URL: https://github.com/apache/beam/pull/11086#issuecomment-597900858
 
 
   @chamikaramj @tvalentyn 
   
   Hey there, I'm working on this improvement. There's some interesting outcomes here:
   
   - For datetime types (date,time,datetime,timestamp), AvroSource returns python types (datetime.date,
datetime.time,datetime.datetime, datetime.datetime) - along with timezone info. I feel that
in this case, Python types are the most desirable. Thoughts? _CustomBQSource is experimental,
so I wouldn't think it's subject to backwards compatibility concerns.
   
   - For bytes, AvroSource is currently returning base64 encoded bytes. I am trying to figure
out how to handle this. For py2, strings and bytes may not be differentiable, so we may not
be able to handle this properly...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message