hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Multi get/put
Date Wed, 06 Aug 2008 20:42:17 GMT
Ning Li wrote:
>> Does you have to do a rewrite of the lucene index at compaction time?  Or
>> just call optimize?  (I suppose its the former if you need to clean up
>> 'References' as per below where you talk of splits)
> What do you mean by "a rewrite of the lucene index"? 

In hbase, on split, daughters hold a reference to either the top or 
bottom half of their parent region.  References are undone by 
compactions; as part of compaction, the part of the parent referenced by 
the daughter gets written out to store files under the daughter.  
Daughters try to undo references as promptly as possible because regions 
with references are not splitable (references to references, and so on, 
would soon become unmanageble).

In your description, you mentioned that daughter regions reference their 
parents' index.  When I said, 'a rewrite of the lucene index', I was 
asking, as per hbase regions, if you followed the model and wrote a new 
lucene index comprised of daughter-only content at compaction time.  Or 
do you just 'optimize' and let the references build up so the daughter 
of a daughter points all the ways up to the parent?

Just wondering.

>> Regards your 'on the other hand' above, thats a good point.  Have you
>> verified that if a regionerver is running on a datanode, that the lucene
>> index is written local?  Would be interesting to know.
> That's HDFS's policy. See HDFS's FSNamesystem.getAdditionalBlock.
Sorry.  Yeah, of course.

So, why do you think it so slow going via HDFS FileSystem when the data 
is local?  Is it the block-orientated access or is there just a high-tax 
going via the HDFS FS interface?


View raw message