hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Date Wed, 14 Jan 2015 00:11:08 GMT
Hi sathya,

I agree with Nik: Ouch.

Do you see this file? It doesn't look right as you mention:
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e

It is 455GB and that times 3 can easily cause the DNs to run out of space.
It would be really useful if you could provide all the configurations that
the RS  is using along with some jstacks. Perhaps just loop curl to grab
the output from the /dump servlet from the RS:

$ (for i in {1..30}; do echo curl http://rs:port/dump ; sleep 10; done) >
rs.dumps.txt

regards,
esteban.



--
Cloudera, Inc.


On Tue, Jan 13, 2015 at 2:53 PM, Sean Busbey <busbey@cloudera.com> wrote:

> what are the configuration settings for
>
> * dfs.data.dir / dfs.datanode.data.dir
> * dfs.datanode.du.reserved
>
> For each of the directories in the first list, can you get the output of
> "tune2fs -l" ?
>
> Can you get us any non-default settings for the hbase cluster and table?
> Are you using compression on the CF?
>
>
> On Tue, Jan 13, 2015 at 1:37 PM, sathyafmt <sathyafmt@gmail.com> wrote:
>
> > Yes, we normally run a 4 node hbase instance with 500G on each of the
> nodes
> > for HDFS. Here's the hadoop fs -ls from a single node instance.
> >
> > sathya
> > =====
> >
> > user@node1:/var/lib$ hadoop fs -ls -R /hbase
> > drwxr-xr-x   - hbase hbase          0 2015-01-05 10:59 /hbase/.tmp
> > drwxr-xr-x   - hbase hbase          0 2015-01-05 10:59 /hbase/WALs
> > drwxr-xr-x   - hbase hbase          0 2015-01-06 13:00
> > /hbase/WALs/node1,60020,1420484330328
> > -rw-r--r--   3 hbase hbase   29728249 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420574422072
> > -rw-r--r--   3 hbase hbase          9 2015-01-06 12:59
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420577990013.meta
> > -rw-r--r--   3 hbase hbase          9 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420578022196
> > drwxr-xr-x   - hbase hbase          0 2015-01-06 11:36 /hbase/archive
> > drwxr-xr-x   - hbase hbase          0 2015-01-05 10:53 /hbase/corrupt
> > drwxr-xr-x   - hbase hbase          0 2014-12-17 06:38 /hbase/data
> > drwxr-xr-x   - hbase hbase          0 2014-12-17 06:45
> /hbase/data/default
> > drwxr-xr-x   - hbase hbase          0 2015-01-05 10:59
> > /hbase/data/default/tsdb
> > drwxr-xr-x   - hbase hbase          0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc
> > -rw-r--r--   3 hbase hbase        283 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc/.tableinfo.0000000001
> > drwxr-xr-x   - hbase hbase          0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tmp
> > drwxr-xr-x   - hbase hbase          0 2015-01-06 11:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71
> > -rwxr-xr-x   3 hbase hbase         50 2014-12-30 16:14
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.regioninfo
> > drwxr-xr-x   - hbase hbase          0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp
> >
> > -rwxr-xr-x   3 hbase hbase 488820965376 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
> >
> > -drwxr-xr-x   - hbase hbase            0 2015-01-05 10:53
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/recovered.edits
> > drwxr-xr-x   - hbase hbase            0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t
> > -rwxr-xr-x   3 hbase hbase       977014 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/0eea561cb7b841a7b7dbb6ead58f4556
> > -rwxr-xr-x   3 hbase hbase       775564 2015-01-05 10:53
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/130b7f0ca2e646e29771367740713911
> > -rwxr-xr-x   3 hbase hbase       216732 2015-01-06 11:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/3bbb590e4f1e4b57a9cca41a80271738
> > drwxr-xr-x   - hbase hbase            0 2015-01-05 13:00
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453
> > -rwxr-xr-x   3 hbase hbase           50 2014-12-30 16:14
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.regioninfo
> > drwxr-xr-x   - hbase hbase            0 2015-01-06 13:16
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp
> > -rwxr-xr-x   3 hbase hbase            0 2015-01-06 13:16
> >
> >
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp/20679e20318c479ca30d980d9d26293e
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067589.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Sean
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message