kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 기준 <0ctopus13pr...@gmail.com>
Subject Re: Question about memory_limit_hard_bytes and
Date Thu, 27 Apr 2017 07:25:38 GMT
Thanks David Alves!!

Have a nice day! :)

2017-04-26 0:26 GMT+09:00 David Alves <davidralves@gmail.com>:

> Hi Jason
>
>   Yeah that's not expected to work.
>   The tserver might accept that config but eventually the block cache will
> consume all available memory up to 4GB and everything else will be starved
> for memory.
>
> -david
>
> On Tue, Apr 25, 2017 at 3:53 AM, 기준 <0ctopus13prime@gmail.com> wrote:
>
>> @David Alves
>>
>> So it is not working if i set memory limit as 4G but set 10G as block
>> cache mb?
>>
>> 2017-04-25 2:45 GMT+09:00 David Alves <davidralves@gmail.com>:
>>
>>> Hi Jason
>>>
>>>   memory_limit_hard_bytes refers to all the memory consumed by the
>>> tablet server. It has an effect in many things but arguably the biggest
>>> effect is that the tablet server will reject all writes when memory
>>> consumption reaches this limit.
>>>   block_cache_capacity_mb dictates how much memory should be saved for
>>> the block cache. The block cache caches blocks that are read from disk,
>>> thus speeding up scans that hit cached data. The block cache also has some
>>> effect on writes, by allowing to cache bloomfilter blocks, which we lookup
>>> to make sure that rows are unique. The page cache caches previously fetched
>>> disk pages, but is managed by the OS and thus out of Kudu's control.
>>>
>>> HTH
>>> David
>>>
>>> On Mon, Apr 24, 2017 at 8:24 AM, Jason Heo <jason.heo.sde@gmail.com>
>>> wrote:
>>>
>>>> Hello.
>>>>
>>>> I'm using Apache Kudu 1.3.0.
>>>>
>>>> I found some explanation in manual and cdh about memory_limit_hard and
>>>> block_cache_capacity But I couldn't understand what are they exactly used
>>>> for, what is stored at block_cache, and what's the meaning of hard limit
of
>>>> memory?
>>>>
>>>> In Kudu Configuration Reference:
>>>>
>>>>    - *memory_limit_hard_bytes
>>>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_memory_limit_hard_bytes>*:
>>>>    Maximum amount of memory this daemon should use, in bytes. A value of
0
>>>>    autosizes based on the total system memory. A value of -1 disables all
>>>>    memory limiting.
>>>>    - *block_cache_capacity_mb
>>>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_block_cache_capacity_mb>*:
>>>>    block cache capacity in MB
>>>>
>>>>
>>>> In Cloudera Manager's help:
>>>>
>>>>    - *memory_limit_hard_bytes*: Maximum amount of memory that the Kudu
>>>>    Tablet Server will use before it starts rejecting all incoming writes.
>>>>    - *block_cache_capacity_mb*: Maximum amount of memory allocated to
>>>>    the Kudu Tablet Server's block cache.
>>>>
>>>> Is hard_limit related with insert/update/delete and is not related to
>>>> performance of read? Does block_cache store tablet data? Then what's the
>>>> relationship with page cache.
>>>>
>>>> After watching Comcast presentation, I was surprised because they are
>>>> using memory_limit_hard=4G, block_cache=512MB.
>>>>
>>>> Regards,
>>>>
>>>> Jason
>>>>
>>>
>>>
>>
>

Mime
View raw message