hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject Re: EndPoint Coprocessor could be dealocked?
Date Wed, 16 May 2012 19:07:46 GMT

 I think we need to look at the base problem that is trying to be solved. 

I mean the discussion on the RPC mechanism. but the problem that the OP is trying to solve
is how to use multiple indexes in a 'query'. 

Note: I put ' ' around query because its a m/r job or a single thread where the user is trying
to get a result set which is a significantly smaller subset, using more than 1 index.

So the idea is to do a quick get() against each index and the result would be a list of row
keys. The next step is to get the intersection(s) quickly (which I proposed), and then you
would just need to do a quick series of get()s  to pull back the list of rows. 

If I understand the OP's problem, its not a co-processor type of problem. 

Its one of where you submit a m/r job. Within your toolRunner, you would actually do the fetches
against the indexes and then build the ultimate result set. then you just need a map job to
take your result set as an input. 

Drawback... if the list of rows is very, very long, you may run out of memory. So you need
to resolve that...
(Which is why I was suggesting on using a temp table and then you can use the rows in the
temp table as input in to your fetch...

While not something I would use for 'real time' its something where I can really shrink the
number of rows you have to fetch for further processing.  So if your full table scan takes
an hour, but we can do N get()s to get the rows in the Index, find the intersection I and
then do I.size() get()s to fetch the data.  This should take much less time. 

Again, I don't see this in a coprocessor based solution, however, the N get()s and intersection
could be done at the start of the job, or could be part of a Map only job.

Kind of an interesting problem...  but if anyone has a large set of data and some time to
play, you will end up solving a problem that you can' do in an RDBMS easily.

On May 16, 2012, at 1:17 PM, Andrew Purtell wrote:

>> On May 16, 2012, at 1:12 AM, fding hbase wrote:
>>> But sadly, HBase ipc doesn't allow coprocessor chaining mechanism...
>>> Someone mentioned on
>>> http://grokbase.com/t/hbase/user/116hrhhf8m/coprocessor-failure-question-and-examples
>>> :
>>> If a RegionObserver issues RPC to another table from any of the hooks that
>>> are called out of RPC handlers (for Gets, Puts, Deletes, etc.), you risk
>>> deadlock. Whatever activity you want to check should be in the same
>>> region as account data to avoid that.
>>> (Or HBase RPC needs to change.)
>>> So, that means, the deadlock is inevitable under current circumstance. The
>>> coprocessors are still limited.
>>> What I'm seeking is possible extensions of coprocessors or workaround for
>>> such situations that extra RPC is needed in the RPC handlers.
> This isn't a limitation, this is a design choice. Such extensions of
> coprocessors most likely won't happen. What a RegionObserver allows
> you to do is exactly this: Intercept and potentially modify lifecycle
> or user operations on that single region alone. If it helps, think of
> each region as its own independent database.
> If you need to take cross-region actions according to some user
> action, then you should be looking first at extending the client, not
> the server.
> Best regards,
>    - Andy
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)

View raw message