trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jackson <jacksontj...@gmail.com>
Subject Re: Interim cache - High CPU usage
Date Tue, 10 Mar 2015 00:57:35 GMT
Personally I'd just avoid the interim cache all together. If you want
tiered storage it is better to do it at the block level. In newer linux
kernels there are a few (I'd use http://en.wikipedia.org/wiki/Bcache). If
not you can compile kernel modules for other (such as flashcache). These
use block-level caching-- so they'll do a better job of caching hot blocks
instead of full http responses-- especially useful if clients abort
downloads soon after starting. In addition this will let your SSD maintain
parts of your files instead of having all or none of it.

On Mon, Jan 12, 2015 at 9:46 AM, Daniel Biazus <daniel.biazus@azion.com>
wrote:

> Hi guys,
>
>     I agree with your approach, I also think that objects are being
> migrated too frequently between the storage and interim cache. However, I
> can't see any workaround regarding this issue, since the interim cache
> doesn't have a configuration that limits the object size on SSD storage,
> like we have in RAM cache, "proxy.config.cache.ram_cache_cutoff".
>     I tried to configure the "proxy.config.cache.interim.migrate_threshold"
> variable to a higher value, something like 20, but it is not enough because
> when a large object (1GB) reaches this threshold, it is copied to interim
> cache, overloading the CPU and IO.
>
> perf record output:
>
>
>
> ​
>
> Does anyone have an insight about this situation ?
>
> Let me know if you guys needs some more info.
>
> Thank You & Best Regards,
>
>
> On Mon, Jan 12, 2015 at 1:49 AM, gang li <portl4t.cn@gmail.com> wrote:
>
>> I don't think it is a good idea to use interim cache if the size of the
>> cache objects are very large. As the ssd disk is only 120GB,  the objects
>> on the HDD will be migrated to the ssd frequently, and the ssd storage is
>> meaningless as it will be overwritten quickly, and this will increase the
>> consumption of cpu and io.
>>
>> Can you give more infos from the perf, such as call graph.
>>
>>
>> On Mon, Jan 12, 2015 at 4:52 AM, Daniel Biazus <daniel.biazus@azion.com>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> We' ve been using ATS as a reverse proxy, and a few week ago We started
>>> to use the interim cache feature in more intense way, caching objects with
>>> the average size of 200 MB and max size of 1 GB.
>>>
>>> *We have ~ 1 TB HDD as a default storage:*
>>>
>>> cat /etc/trafficserver/storage.config
>>>
>>> # ATS - Storage
>>> /dev/sda6 volume=1
>>>
>>> *And also, a 120 GB SDD storage as interim cache:*
>>>
>>> LOCAL proxy.config.cache.interim.storage STRING /dev/sdc1
>>>
>>> After 20 ~ 30 minutes in production with this configuration, We could
>>> notice a sudden CPU high usage, increasing up to 65 %, considering that our
>>> regular usage is 10 %. However the throughput still stable in 250 Mbps per
>>> box.
>>>
>>> We've found the following behavior using the perf top tool:
>>>
>>>    88.18%  traffic_server      [.]
>>> _Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread
>>>      0.32%  traffic_server      [.]
>>> _ZN10NetHandler12mainNetEventEiP5Event
>>>      0.30%  [kernel]             [k] update_sd_lb_stats
>>>      0.29%  [e1000e]           [k] e1000e_check_ltr_demote
>>>      0.25%  [kernel]             [k] __ticket_spin_lock
>>>      0.24%  traffic_server      [.] _ZN7EThread13process_eventEP5Eventi
>>>      0.21%  [kernel]             [k] timerqueue_add
>>>      0.17%  libc-2.12.so       [.] epoll_wait
>>>      0.17%  libpcre.so.0.0.   [.] 0x00000000000100dd
>>>      0.14%  [kernel]             [k] __schedule
>>>
>>> 1) This behavior is easily reproduced caching* large objects with
>>> interim cache active*.
>>> 2) With interim cache *disabled*, this behavior* is not reproduced.*
>>>
>>>  As you can see, at the perf top output, the write_to_net_*io *function
>>> is responsible for this heavy CPU usage. We would like to hear of you guys,
>>> if anyone has faced a issue like that, or if you have any clues about this
>>> possible bug.
>>>
>>> Thanks & Regards,
>>>
>>> --
>>>
>>> Daniel Biazus
>>> infrastructure Engineering
>>> Azion Technologies
>>> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
>>> Miami, USA +1 305 704 8816
>>>
>>> Quaisquer informações contidas neste e-mail e anexos podem ser
>>> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
>>> de utilização deste documento depende de autorização do emissor, sujeito
as
>>> penalidades cabíveis.
>>>
>>> Any information in this e-mail and attachments may be confidential and
>>> privileged, protected by legal confidentiality. The use of this document
>>> require authorization by the issuer, subject to penalties.
>>>
>>>
>>
>
>
> --
>
> Daniel Biazus
> infrastructure Engineering
> Azion Technologies
> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
> Miami, USA +1 305 704 8816
>
> Quaisquer informações contidas neste e-mail e anexos podem ser
> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
> de utilização deste documento depende de autorização do emissor, sujeito as
> penalidades cabíveis.
>
> Any information in this e-mail and attachments may be confidential and
> privileged, protected by legal confidentiality. The use of this document
> require authorization by the issuer, subject to penalties.
>
>

Mime
View raw message