kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Berkeley <wdberke...@cloudera.com>
Subject Re: CDH 5.5 - Kudu error not enough space remaining in buffer for op
Date Wed, 18 May 2016 03:32:44 GMT
Hi Abhi.

I believe that error is actually coming from the client, not the server.
See e,g,
https://github.com/apache/incubator-kudu/blob/master/src/kudu/client/batcher.cc#L787
(NB
that link is to master branch not the exact release you are using).

If you look around there, you'll see that the max is set by something
called max_buffer_size_, which appears to be hardcoded to 7 * 1024 * 1024
bytes = 7MiB (and this is consistent with 6.96 + 0.0467 > 7).

I think the simple workaround would be to do the CTAS as a CTAS + insert as
select. Pick a condition that bipartitions the table, so you don't get
errors trying to double insert rows.

-Will

On Tue, May 17, 2016 at 4:45 PM, Abhi Basu <9000revs@gmail.com> wrote:

> What is the limit of columns in Kudu?
>
> I am using 1000 gen dataset, specifically the chr22 table which has
> 500,000 rows x 1101 columns. This table has been built In Impala/HDFS. I am
> trying to create a new Kudu table as select from that table. I get the
> following error:
>
> Error while applying Kudu session.: Incomplete: not enough space remaining
> in buffer for op (required 46.7K, 6.96M already used
>
> When looking at http://pcsd-cdh2.local.com:8051/mem-trackers, I see the
> following. What configuration needs to be tweaked?
>
>
> Memory usage by subsystem
> IdParentLimitCurrent ConsumptionPeak consumption
> root none 50.12G 4.97M 6.08M
> block_cache-sharded_lru_cache root none 937.9K 937.9K
> code_cache-sharded_lru_cache root none 1B 1B
> server root none 2.3K 201.4K
> tablet-00000000000000000000000000000000 server none 530B 200.1K
> MemRowSet-6 tablet-00000000000000000000000000000000 none 265B 265B
> txn_tracker tablet-00000000000000000000000000000000 64.00M 0B 28.5K
> DeltaMemStores tablet-00000000000000000000000000000000 none 265B 87.8K
> log_block_manager server none 1.8K 2.7K
>
> Thanks,
> --
> Abhi Basu
>

Mime
View raw message