gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Dynamically generating HBase columns
Date Thu, 26 Feb 2015 20:16:56 GMT
Hi Alfonso,

On Tue, Feb 24, 2015 at 10:27 PM, <dev-digest-help@gora.apache.org> wrote:

> In my use cases I always need a mix between static and dynamic columns.
> In my first week I tried to mix a Map over a column family overlapped with
> static columns. Didn't work because Gora was not prepared for that (and
> indeed needs thinking about it further).

Yeah. I've logged the following focus to deal with it

> What I do is separate the static columns in one column family (or serveral)
> from the dynamic stuff (that goes in a map). One Map is mapped to one
> column family in which each column:value is key=>value in the map.
> I have several maps depending on my needs, but can be just one big one with
> key=column.

Can you please show this graphically so I am absolutely clear on what you
are doing?

> What I don't fully understand is the timestamp you talk about, since we
> don't handle HBase timestamps. Do you specifically need it?

Yes, please read comment on GORA-413

> I'm not quite sure if I answer you :S

We will clarify it soon. Don;t worry ;)

> Something important to ask is much columns will you store in the column
> family?

Well dynamic columns will be added with every incoming chunk of data.

> Since we removed the StateManager, when you modify a map it deletes the
> column familiy and sends all the data again to be written (
> https://github.com/apache/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L289
> ),
> so adding/removing just one column can be quite killing when persisting
> several huge maps. About what volume and write pattern are we talking?

The volume of data will not be so large however it is concerning that
entire column families are deleted and re-written. It seems like a waste of
time and resources which we should address in an effort to make this a more
efficient process.
Thanks, lets take the discussion over to GORA-413

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message