kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Heo <jason.heo....@gmail.com>
Subject Re: How to flush `block_cache_capacity_mb` easily?
Date Tue, 11 Apr 2017 01:14:51 GMT
Great. I would be appreciated it if you guide me how can I contribute it.
Then I'll try in my spare time.

2017-04-11 7:46 GMT+09:00 Todd Lipcon <todd@cloudera.com>:

> On Sun, Apr 9, 2017 at 6:38 PM, Jason Heo <jason.heo.sde@gmail.com> wrote:
>
>> Hi Todd.
>>
>> I hope you had a good weekend.
>>
>> Exactly, I'm testing the latency of cold-cache reads from SATA disks and
>> performance of difference schema designs as well.
>>
>> We currently using Elasticsearch for a analytic service. ES has a "clear
>> cache API" feature, it makes me easy to test.
>>
>>
> Makes sense. I don't think it would be particularly difficult to add such
> an API. Any interest in contributing a patch? I'm happy to point you in the
> right direction, if so.
>
> -Todd
>
>
>> 2017-04-08 5:05 GMT+09:00 Todd Lipcon <todd@cloudera.com>:
>>
>>> Hey Jason,
>>>
>>> Can I ask what the purposes of the testing is?
>>>
>>> One thing to note is that we're currently leaving a fair bit of
>>> performance on the table for cold-cache reads from spinning disks. So, if
>>> you find that the performance is not satisfactory, it's worth being aware
>>> that we will likely make some significant improvements in this area in the
>>> future.
>>>
>>> https://issues.apache.org/jira/browse/KUDU-1289 has some details.
>>>
>>> -Todd
>>>
>>> On Fri, Apr 7, 2017 at 8:44 AM, Dan Burkert <danburkert@apache.org>
>>> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> There is no command to have Kudu evict its block cache, but restarting
>>>> the tablet server process will have that effect.  Ideally all written data
>>>> will be flushed before the restart, otherwise startup/bootstrap will take
a
>>>> bit longer. Flushing typically happens within 60s of the last write.
>>>> Waiting for flush and compaction is also a best-practice for read-only
>>>> benchmarks.  I'm not sure if someone else on the list has an easier way of
>>>> determining when a flush happens, but I typically look at the 'MemRowSet'
>>>> memory usage for the tablet on the /mem-trackers HTTP endpoint; it should
>>>> show something minimal like 256B if it's fully flushed and empty.  You can
>>>> also see details about how much memory is in the block cache on that page,
>>>> if that interests you.
>>>>
>>>> - Dan
>>>>
>>>> On Thu, Apr 6, 2017 at 11:23 PM, Jason Heo <jason.heo.sde@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> I'm using Apache Kudu 1.2 on CDH 5.10.
>>>>>
>>>>> Currently, I'm doing a performance test of Kudu.
>>>>>
>>>>> Flushing OS Page Cache is easy, but I don't know how to flush
>>>>> `block_cache_capacity_mb` easily.
>>>>>
>>>>> I currently execute SELECT statement over a unnecessarily table to
>>>>> evict cached block of testing table.
>>>>>
>>>>> It is cumbersome, so I'd like to know is there a command for flushing
>>>>> block caches (or another kudu's caches which I don't know yet)
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Regards,
>>>>> Jason
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message