phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantinos Kougios <kostas.koug...@googlemail.com>
Subject issue when creating index for a large table
Date Tue, 29 Sep 2015 09:10:14 GMT
I got a table "words" which is 200GB on a not-so-fast cluster, 7200rpm 
disks.

0: jdbc:phoenix:nn.lan> create index IX_WORDS on words(word);
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=80ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 80 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=142ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 142 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=141ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 141 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=61ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 61 - final failure

It eventually fails the creation of the index. Is there a config I 
should alter to fix this?



On the server log there is no exception:

2015-09-29 10:07:47,193 INFO  [MemStoreFlusher.1] 
regionserver.DefaultStoreFlusher: Flushed, sequenceid=846905, 
memsize=128.2 M, hasBloomFilter=true, into tmp file 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/.tmp/3fee1d78302146ccac035138a498a966
2015-09-29 10:07:47,204 INFO  [MemStoreFlusher.1] regionserver.HStore: 
Added 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/0/3fee1d78302146ccac035138a498a966,

entries=786646, sequenceid=846905, filesize=13.4 M
2015-09-29 10:07:47,208 INFO  [MemStoreFlusher.1] regionserver.HRegion: 
Finished memstore flush of ~128.19 MB/134418456, currentsize=11.45 
MB/12001160 for region 
IX_WORDS,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00,1443516250552.e4bcc5bcda8227204db81eced37b5546.

in 1972ms, sequenceid=846905, compaction requested=true
2015-09-29 10:07:53,630 INFO  [sync.3] wal.FSHLog: Slow sync cost: 143 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:56,807 INFO  [sync.3] wal.FSHLog: Slow sync cost: 113 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:56,808 INFO  [sync.2] wal.FSHLog: Slow sync cost: 115 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:57,225 INFO  [sync.4] wal.FSHLog: Slow sync cost: 103 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,693 INFO  [sync.3] wal.FSHLog: Slow sync cost: 939 
ms, current pipeline: [192.168.0.31:50010]
*2015-09-29 10:07:58,693 INFO  [sync.4] wal.FSHLog: Slow sync cost: 939 
ms, current pipeline: [192.168.0.31:50010]**
*2015-09-29 10:07:58,693 INFO  [sync.0] wal.FSHLog: Slow sync cost: 938 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,693 INFO  [sync.2] wal.FSHLog: Slow sync cost: 940 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,694 INFO  [sync.1] wal.FSHLog: Slow sync cost: 938 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:59,484 INFO  [sync.4] wal.FSHLog: Slow sync cost: 147 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,116 INFO  [sync.3] wal.FSHLog: Slow sync cost: 131 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,117 INFO  [sync.2] wal.FSHLog: Slow sync cost: 133 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,654 INFO  [sync.2] wal.FSHLog: Slow sync cost: 127 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:01,128 INFO  [sync.0] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:01,128 INFO  [sync.1] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:02,988 INFO 
[regionserver/d3.lan/192.168.0.31:16020.logRoller] wal.FSHLog: Rolled 
WAL 
/hbase/WALs/d3.lan,16020,1443514600886/d3.lan%2C16020%2C1443514600886.default.1443517654445

with entries=23584, filesize=121.87 MB; new WAL 
/hbase/WALs/d3.lan,16020,1443514600886/d3.lan%2C16020%2C1443514600886.default.1443517682939
2015-09-29 10:08:08,122 INFO  [MemStoreFlusher.1] regionserver.HRegion: 
Started memstore flush for 
IX_WORDS,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00,1443516250552.e4bcc5bcda8227204db81eced37b5546.,

current region memstore size 128.00 MB, and 1/1 column families' 
memstores are being flushed.
2015-09-29 10:08:10,819 INFO  [MemStoreFlusher.1] 
regionserver.DefaultStoreFlusher: Flushed, sequenceid=859636, 
memsize=128.0 M, hasBloomFilter=true, into tmp file 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/.tmp/197a91b7e345448a9905576f0e66e411
2015-09-29 10:08:10,839 INFO  [MemStoreFlusher.1] regionserver.HStore: 
Added 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/0/197a91b7e345448a9905576f0e66e411,

entries=785829, sequenceid=859636, filesize=13.4 M
2015-09-29 10:08:10,850 INFO  [MemStoreFlusher.1] regionserver.HRegion: 
Finished memstore flush of ~128.02 MB/134242872, currentsize=15.92 
MB/16692544 for region 
IX_WORDS,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00,1443516250552.e4bcc5bcda8227204db81eced37b5546.

in 2728ms, sequenceid=859636, compaction requested=true
2015-09-29 10:08:11,087 INFO  [sync.3] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]
*2015-09-29 10:08:11,087 INFO  [sync.4] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]**
*2015-09-29 10:08:11,087 INFO  [sync.0] wal.FSHLog: Slow sync cost: 118 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:11,088 INFO  [sync.1] wal.FSHLog: Slow sync cost: 119 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:11,089 INFO  [sync.2] wal.FSHLog: Slow sync cost: 116 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:11,912 INFO  [sync.1] wal.FSHLog: Slow sync cost: 125 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:11,914 INFO  [sync.0] wal.FSHLog: Slow sync cost: 127 
ms, current pipeline: [192.168.0.31:50010]

Thanks

Mime
View raw message