hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From y_823...@tsmc.com
Subject Re: Save data to HBase
Date Wed, 02 Sep 2009 08:02:35 GMT
                                                                                          
                                                           
                      y_823910@tsmc.com                                                  
                                                            
                                               To:      hbase-user@hadoop.apache.org     
                                                            
                      2009/09/02 01:52         cc:      (bcc: Y_823910/TSMC)             
                                                            
                      PM                       Subject: Re: Save data to HBase           
                                                            
                      Please respond to                                                  
                                                            
                      hbase-user                                                         
                                                            
                                                                                         
                                                            
                                                                                         
                                                            







Hi,
I just use 3 core2 pc as RS with (1G,1G,2G RAM).
Right! I didn't change anything in different insert runs.
These warn point to save2HBaseThread.putFRPF(save2HBaseThread.java:105)
Line 105 is the code table1 = new HTable("MyTable");
It's still working after following WARN, but if I continue to do the
following jobs.
These warning raised randomly.If you continue to do jobs, it will fail to
put data.
I guess 200MB is not a large dataset,but it took me so much time to put it.
Any suggestion?
Thanks

Fleming

WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum
servers, currently=10.42.253.86:2181,10.42.253.182:2181,10.42.254.37:2181
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
      at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:522)
      at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:342)

      at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:365)

      at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:478)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:846)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:528)

      at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:123)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:78)
      at save2HBaseThread.putFRPF(save2HBaseThread.java:105)
      at save2HBaseThread.run(save2HBaseThread.java:66)





                      "Jonathan Gray"

                      <jlist@streamy.co        To:
hbase-user@hadoop.apache.org

                      m>                       cc:      (bcc:
Y_823910/TSMC)

                                               Subject: Re: Save data to
HBase

                      2009/09/02 01:17

                      PM

                      Please respond to

                      hbase-user







Once the JVM consumes memory it generally will not release it back to the
OS (so you will see it near-full).

How much heap are you giving the RS?

The issue you're having seems rather odd... Nothing at all changes between
the different insert runs?  Instantiating a new HTable should not be
creating a /hbase node.

On Tue, September 1, 2009 8:37 pm, y_823910@tsmc.com wrote:
> Hi,
>
>
> I guess Creation of /hbase happen after running table1 = new
> HTable("MyTable");
> Right, that warning showed up in the client.There doesn't keep any log in
> that region server. When ever I finish to put my data, I would count it
by
> shell. I put the first 10 files(10M) in five threads. It worked pretty
> well. After taking a minutes break, I put next 30 files.It still worked
> very quickly. When I count it, I found it became very very slow.
> By checking that region server,I found that region server taken so much
> RAM(almost on the top),
> and seems not to release it. Maybe I should lower the flush
threshold.(how
> to do it ? ^_^) If I don't take a break and continue to put data, that
> warning raised in the client. So, I wonder that region server is too busy
> to serve the following client request. If a client can't reach that busy
> sever, it would be better to raise some exception that I can catch and
try
> to handle it. Thanks
>
>
> Fleming
>
>
>
>
>
>
> Jonathan Gray
> <jlist@streamy.co        To:      hbase-user@hadoop.apache.org
> m>                       cc:      (bcc: Y_823910/TSMC) Subject: Re: Save
> data to HBase 2009/09/02 10:22
> AM
> Please respond to
> hbase-user
>
>
>
>
>
>
> There's no reason to use the write buffer if you are only inserting one
> row and then immediately flushing.  This code does the exact same thing
as
> yours:
>
>
> table1 = new HTable("MyTable"); Put p = new Put(...);
> p.add(...); table1.put(p);
>
> That's it.  You would only use the write-buffering if you wanted to send
> multiple Puts at once.
>
> As far as the table only having one region, yes this limits all work on
> that table to a single node.  However with 200MB of data, you are smaller
> than the default region size of 256MB.  You can lower that split
threshold
> or you can manually split in the shell or using HBaseAdmin.
>
> Not sure about that ZooKeeper warning... Creation of /hbase should
> happen in the master at boot-time, not in a client running inserts.  Do
you
> have anymore detail?  That warning showed up in the client? Anything in
> the regionserver?  Can you use the shell?
>
> JG
>
>
> y_823910@tsmc.com wrote:
>> Hi,
>>
>>
>> I want to put only 200MB data to a HBase table.
>> Just use a single thread to put 1MB data in a time.
>> So, there are 200 job in turn.
>> It occured WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase,
>> while it run to around the 20th job. I found that table has just one
>> regin. I wonder whether these tight job will make that region server
>> choke. About the region split, I want to know when the table's region
>> will be splitted. Is its threshold setting in hbase-default.xml?
>> Following are a snippet of my code about putting my data to hbase,
>>
> anything
>> can improve it?
>>
>> table1 = new HTable("MyTable"); table1.setAutoFlush(false);
>> table1.setWriteBufferSize(1024*1024*12);
>>
>> Put p = new Put(Bytes.toBytes(rs.getString(KeyColumn)));
>> p.add(Bytes.toBytes(FamColumn),
>> Bytes.toBytes("content"),Bytes.toBytes(json));
>> table1.put(p);
>>
>> table1.flushCommits(); table1.close();
>>
>> My cluster is only 3 PC with (1G,1G,2G RAM ,2 core CPU )
>> 3 region server
>> 3 ZK quorum server
>> We'll enlarge our cluster to 12PCs with 8G RAM.
>>
>>
>> Thanks
>>
>>
>> Fleming
>>
>>
>>
>>
>>
>
>> Amandeep Khurana
>>
>
>> <amansk@gmail.com        To:
>>
> hbase-user@hadoop.apache.org
>
>>> cc:      (bcc:
>>>
> Y_823910/TSMC)
>
>
>> Subject: Re: Save data to
>>
> HBase
>
>
>> 2009/09/02 03:07
>>
>
>> AM
>>
>
>> Please respond to
>>
>
>> hbase-user
>
>>
>
>>
>
>>
>>
>>
>>
>> What do you mean by quick way? You can use the api to put data into it
>> through a standalone java program or you can use a MR job to do it..
>>
>>
>> On Tue, Sep 1, 2009 at 2:48 AM, <y_823910@tsmc.com> wrote:
>>
>>
>>> Hi there,
>>>
>>>
>>> Any quick way that I can save my data(12G) to HBase?
>>> Thanks
>>>
>>>
>>> Fleming
>>>
>>>
>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
> -------------------------------------------------------------------------
> --
>
>
>> TSMC PROPERTY
>>
>
>> This email communication (and any attachments) is proprietary
>>
> information
>> for the sole use of its
>
>> intended recipient. Any unauthorized review, use or distribution by
> anyone
>> other than the intended
>
>> recipient is strictly prohibited.  If you are not the intended
> recipient,
>> please notify the sender by
>
>> replying to this email, and then delete this email and any copies of it
>>
>
>> immediately. Thank you.
>
>>
> -------------------------------------------------------------------------
> --
>
>
>>
>>
>>
>
>
>
>
> -------------------------------------------------------------------------
> --
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary information
> for the sole use of its intended recipient. Any unauthorized review, use
or
> distribution by anyone other than the intended recipient is strictly
> prohibited.  If you are not the intended recipient, please notify the
> sender by replying to this email, and then delete this email and any
> copies of it immediately. Thank you.
> -------------------------------------------------------------------------
> --
>
>
>
>
>






---------------------------------------------------------------------------

                                                         TSMC PROPERTY

 This email communication (and any attachments) is proprietary information

 for the sole use of its

 intended recipient. Any unauthorized review, use or distribution by anyone

 other than the intended

 recipient is strictly prohibited.  If you are not the intended recipient,

 please notify the sender by

 replying to this email, and then delete this email and any copies of it

 immediately. Thank you.


---------------------------------------------------------------------------








 --------------------------------------------------------------------------- 
                                                         TSMC PROPERTY       
 This email communication (and any attachments) is proprietary information   
 for the sole use of its                                                     
 intended recipient. Any unauthorized review, use or distribution by anyone  
 other than the intended                                                     
 recipient is strictly prohibited.  If you are not the intended recipient,   
 please notify the sender by                                                 
 replying to this email, and then delete this email and any copies of it     
 immediately. Thank you.                                                     
 --------------------------------------------------------------------------- 




Mime
View raw message