trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Veiko Kukk <veiko.k...@gmail.com>
Subject Re: Understanding ATS memory usage
Date Tue, 23 Jan 2018 11:53:24 GMT
Hi,

I should have noted before that, during that timeframe, there is no higher
disk activity than on average. No higher load, no disk latency, no cpu
load. Nothing abnormal except slow ATS.
ATS is running on CentOS 7 directly on hardware dedicated server.

Dirty pages related config that's been always there for that server:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 40
vm.swappiness = 0

# free -m
                   total           used           free          shared
buff/cache   available
Mem:         128831       27841        2436        4331       98554
 95722
Swap:          4095           0        4095

As you see, there is lot of available memory.

I don't see how writing dirty pages could slow down ATS when there is no
indication of excessive load on any of the system resources.
And this strange regularity: every ~ 50 minutes this happens. Like some
regular (cronjob like) task is being ran inside ATS that delays all other
tasks.

ATS is using 9TB raw partition if that information might be relevant.

Could you point me to the documentation dealing with dir entry sync periods
of ATS?


-- 
Veiko



2018-01-23 12:12 GMT+02:00 Leif Hedstrom <zwoop@apache.org>:

>
>
> On Jan 23, 2018, at 7:36 PM, Veiko Kukk <veiko.kukk@gmail.com> wrote:
>
> Hi again,
>
> During that mysterious task that happens after ~ 50-51 minutes causes
> requests/responses to slow down very much, even time out.
> Requests that usually take few hundred milliseconds are now taking over
> 30s and timing out. This happens only during that time when memory
> consumption is suddenly dropped by ATS. Happens for both bypassed urls and
> for hits.
> ATS version is 7.1.1 and this looks like serious bug for me.
>
>
>
> That sounds suspiciously like kernel paging activity, maybe it’s spending
> that time dumping flushing dirty pages?  Maybe transparent huge pages ? Or
> tweak the sysctl’s for dirty page ratios?
>
> The other thing to possibly look at is the dir entry sync periods of ATS.
> Whenever we sync those to disk, we consume both more memory and more disk
> I/O, and maybe you are putting too much pressure on the VM (i.e. maybe you
> need to turn down the RAM cache or tweak the amount of directory entries
> you have).
>
> — Leif
>
> E.g.
>
> https://lonesysadmin.net/2013/12/22/better-linux-disk-
> caching-performance-vm-dirty_ratio/
>
>
>

Mime
View raw message