trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Birdsall <dave.birds...@esgyn.com>
Subject RE: avoiding a JNI call to getHbaseTableInfo?
Date Thu, 06 Aug 2015 00:24:22 GMT
Hi,

Actually I wasn't saying it was in the TDB, but that would be the place to
put it if it isn't there already. Someone more familiar with HBase scan
nodes might know off the top of their head. If not I can take a peek
tomorrow.

Dave

-----Original Message-----
From: Eric Owhadi [mailto:eric.owhadi@esgyn.com]
Sent: Wednesday, August 5, 2015 4:34 PM
To: dev@trafodion.incubator.apache.org
Subject: Re: avoiding a JNI call to getHbaseTableInfo?

That is what I am trying to get: the blocksize from the TDB... still looking
where that is :-)... Now that I know blocksize is there, forget about the
other question, browsing the code I only saw the runtime call, and was
afraid the data was not in TDB, but look like you are saying it is, so this
is perfect, I just have to find it...

On Wed, Aug 5, 2015 at 6:28 PM, Dave Birdsall <dave.birdsall@esgyn.com>
wrote:

> Hi,
>
> Just want to make sure I understood Eric's question. Eric, were you
> considering a run-time call to get this information?
>
> If so, as Qifan points out, this information is available to the compiler.
> We just would have to get it to the run-time somehow. Usual route is
> through the appropriate TDB generated by the compiler and read by the
> executor.
>
> Dave
>
> -----Original Message-----
> From: Qifan Chen [mailto:qifan.chen@esgyn.com]
> Sent: Wednesday, August 5, 2015 4:04 PM
> To: dev <dev@trafodion.incubator.apache.org>
> Subject: Re: avoiding a JNI call to getHbaseTableInfo?
>
> Hi Eric,
>
> You can set the CQD Hbase_index_level to a positive integer value
> (such as
> 2) to completely bypass the JNI call.  Note that the JNI call also
> reads in the index level, which is a function of the Hfiles in use.
>
> Since we cache NATable,  which is  an in-memory representation of a
> HBase table, the JNI cost is only incurred once for all references to
> a table, per SQL session.  If the table is redefined, we will re-read
> the definition and call the JNI again.
>
> Thanks --Qifan
>
> On Wed, Aug 5, 2015 at 5:47 PM, Eric Owhadi <eric.owhadi@esgyn.com> wrote:
>
> > For the small scanner feature, I first hardcoded the default
> > HBaseBlock size (64KB) in the check to see if small scanner is good
> > candidate to enable or not.
> >
> > Then struggling with the regression tests and the modifications
> > needed to pass, I realized that it is better to have a feature
> > totally completed to avoid the cost of dealing with regression every
> > time you want to improve it.
> >
> > So I am trying to remove the hardcode for HBase block size, and
> > realized there is a function getHbaseTableInfo that will do the job
> > of getting HBase block size and index level. BUT, this function is
> > doing a JNI call to the HBase layer. Look like expensive to call for
> > every scan query… will likely lose the gain of small scanner if I
> > put this call in the middle.
> >
> > I was wondering if this Hbase block size per table is not already
> > cached in memory so that I can access it without fear of performance
> > cost? I guess it is table meta data, and I think this is supposed to
> > already be cached if I recall a previous email thread?
> >
> > Thanks in advance for the help,
> >
> > Eric
> >
>
>
>
> --
> Regards, --Qifan
>

Mime
View raw message