beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía (JIRA) <>
Subject [jira] [Commented] (BEAM-2955) Create a Cloud Bigtable HBase connector
Date Wed, 13 Sep 2017 20:05:00 GMT


Ismaël Mejía commented on BEAM-2955:

This is a great idea [~sduskis]. I will be pro (1) but I can understand if the current BigtableIO
maintainers prefer (2). If we go the io/hbase route I would really like that we share as much
code as possible, and of course we can refactor it freely to adapt the other client.
About the missing parts, I added Dynamic Work Rebalancing to it recently (not sure if this
is what you refer to). And I agree that size estimation should be separated. I am OOO until
beginning of october, but you can count on me for anything related to this.

> Create a Cloud Bigtable HBase connector
> ---------------------------------------
>                 Key: BEAM-2955
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-gcp
>            Reporter: Solomon Duskis
>            Assignee: Chamikara Jayalath
> The Cloud Bigtable (CBT) team has had a Dataflow connector maintained in a different
repo for awhile. Recently, we did some reworking of the Cloud Bigtable client that would allow
it to better coexist in the Beam ecosystem, and we also released a Beam connector in our repository
that exposes HBase idioms rather than the Protobuf idioms of BigtableIO.  More information
about the customer experience of the HBase connector can be found here: [].
> The Beam repo is a much better place to house a Cloud Bigtable HBase connector.  There
are a couple of ways we can implement this new connector:
> # The CBT connector depends on artifacts in the io/hbase maven project.  We can create
a new extend HBaseIO for the purposes of CBT.  We would have to add some features to HBaseIO
to make that work (dynamic rebalancing, and a way for HBase and CBT's size estimation models
to coexist)
> # The BigtableIO connector works well, and we can add an adapter layer on top of it.
 I have a proof of concept of it here: [].
> # We can build a separate CBT HBase connector.
> I'm happy to do the work.  I would appreciate some guidance and discussion about the
right approach.

This message was sent by Atlassian JIRA

View raw message