kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 기준 <0ctopus13pr...@gmail.com>
Subject Re: Question about memory_limit_hard_bytes and
Date Tue, 25 Apr 2017 10:53:16 GMT
@David Alves

So it is not working if i set memory limit as 4G but set 10G as block cache
mb?

2017-04-25 2:45 GMT+09:00 David Alves <davidralves@gmail.com>:

> Hi Jason
>
>   memory_limit_hard_bytes refers to all the memory consumed by the tablet
> server. It has an effect in many things but arguably the biggest effect is
> that the tablet server will reject all writes when memory consumption
> reaches this limit.
>   block_cache_capacity_mb dictates how much memory should be saved for the
> block cache. The block cache caches blocks that are read from disk, thus
> speeding up scans that hit cached data. The block cache also has some
> effect on writes, by allowing to cache bloomfilter blocks, which we lookup
> to make sure that rows are unique. The page cache caches previously fetched
> disk pages, but is managed by the OS and thus out of Kudu's control.
>
> HTH
> David
>
> On Mon, Apr 24, 2017 at 8:24 AM, Jason Heo <jason.heo.sde@gmail.com>
> wrote:
>
>> Hello.
>>
>> I'm using Apache Kudu 1.3.0.
>>
>> I found some explanation in manual and cdh about memory_limit_hard and
>> block_cache_capacity But I couldn't understand what are they exactly used
>> for, what is stored at block_cache, and what's the meaning of hard limit of
>> memory?
>>
>> In Kudu Configuration Reference:
>>
>>    - *memory_limit_hard_bytes
>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_memory_limit_hard_bytes>*:
>>    Maximum amount of memory this daemon should use, in bytes. A value of 0
>>    autosizes based on the total system memory. A value of -1 disables all
>>    memory limiting.
>>    - *block_cache_capacity_mb
>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_block_cache_capacity_mb>*:
>>    block cache capacity in MB
>>
>>
>> In Cloudera Manager's help:
>>
>>    - *memory_limit_hard_bytes*: Maximum amount of memory that the Kudu
>>    Tablet Server will use before it starts rejecting all incoming writes.
>>    - *block_cache_capacity_mb*: Maximum amount of memory allocated to
>>    the Kudu Tablet Server's block cache.
>>
>> Is hard_limit related with insert/update/delete and is not related to
>> performance of read? Does block_cache store tablet data? Then what's the
>> relationship with page cache.
>>
>> After watching Comcast presentation, I was surprised because they are
>> using memory_limit_hard=4G, block_cache=512MB.
>>
>> Regards,
>>
>> Jason
>>
>
>

Mime
View raw message