hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Sechrist <ssechr...@gmail.com>
Subject Re: disable table, alter table, and merge regions
Date Wed, 01 Dec 2010 19:35:02 GMT
>From what I remember on our experience (We were either on 0.89.20100614 or
0.89.20100921):

1. I don't think we shut down Zookeeper, just Master and RegionServers. In
fact, the merge utility checks to make sure the master isn't running before
performing the merge (you'll see something like "Exception: master not
running" in the output - that is supposed to happen).

2. Yep, we used that merge tool.

-Sean

On Wed, Dec 1, 2010 at 1:39 PM, Abhijit Pol <apol@rocketfuel.com> wrote:

> Thanks Sean for detailed reply. This is very useful.
>
> I think we can afford to shut down cluster for few hours in the night and
> probably do this in batches. Few questions to use your experience on this:
>
> 1. You are referring to Shut down of HBase to bring down RS, Master
> and Zookeeper? I was getting some errors test merging when all three were
> down.
>
> 2.  Just to make sure this is the merge utility you used?
> $HBASE_HOME/bin/hbase org.apache.hadoop.hbase.util.Merge <tbl_name>
> <region_1> <region_2>
>
>
>
>
> Few follow-up questions
>
>
> On Wed, Dec 1, 2010 at 6:09 AM, Sean Sechrist <ssechrist@gmail.com> wrote:
>
>> Hey Abhijit,
>>
>> We ran into this same issue a while back, and here is what we did (and it
>> seemed to work ok):
>>
>> 1. Went onto to the HBase web ui for our biggest table, and grabbed all of
>> the region names (they appear in order on that page). Saved the region
>> names
>> to a text file.
>> 2. Wrote a shell script to run the hbase merge tool on every pair of
>> regions
>> in that file.
>> 3. Shut down HBase.
>> 4. Run that shell script. It went at about 50 merges/hour on our 5 node
>> cluster.
>> 5. Start HBase. When it went back up we saw that our region count was
>> about
>> 1500 regions, down from almost 3000.
>>
>> So this would only work if you can take down HBase for a decent amount of
>> time.
>>
>> I wonder if you could alternatively, run an Export job and an Import job
>> of
>> your table. Do those preserve the regions, or could you use it to bring
>> down
>> the number of regions?
>>
>> -Sean
>>
>> On Wed, Dec 1, 2010 at 2:24 AM, Abhijit Pol <apol@rocketfuel.com> wrote:
>>
>> > We have HBase cluster which was peacefully (acceptable throughput and
>> > latencies) serving for about a month (we are using 0.89.20100926
>> version)
>> >
>> > This morning we wanted to set TTL to value smaller than default and Mr.
>> > Murphy struck.
>> >
>> > (A) We disabled and altered table with desired TTL value (using shell).
>> > Alerting one property (in this case TTL) reseted all other
>> > table properties to default values. For example version was reset to 3,
>> > compression was reset to NONE etc. [I think this is known issue with
>> open
>> > Jira]
>> >
>> > (B) We wanted to go back to previous table properties. Now after
>> multiple
>> > retires we were not able to disable table (even restarting cluster
>> didn't
>> > helped). Most likely if clients are hitting hard (in this case ~10k qps)
>> on
>> > HBase table, it takes forever to disable a table.
>> >
>> > So we stopped all clients and then were able to disable table and
>> altered
>> > table properties to desired values.
>> >
>> > (C) Due to compression was reset to NONE and version was reset to 3 for
>> > good
>> > 10-12hrs, the total number of regions tripled and load (#regions/RS)
>> > increased from 100 to 300. After first major_compaction, compression,
>> > version 1, and new lower TTL became effective and we were back to
>> original
>> > HDFS footprint and have bunch of small regions.
>> >
>> > What will trigger merging of these regions? the tool for merging does
>> not
>> > seem to work or even if it does, it can only do two regions at-a-time.
>> >
>> > Any suggestions on how we can reduce number of regions and bring load
>> back
>> > to where it was before?
>> >
>> >
>> > Thanks,
>> > --Abhi
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message