I tried the stats_over_http plugin today and it's behaving the same.
memory/hdrHeap and memory/hdrStrHeap are increasing on every requests.
Will file an issue. Not only a Virtual Machine issue.
Tried to get jemalloc to help us but I believe we failed. No output.
How to reproduce?
#!/bin/bash
while true; do
curl http://<ip>/_stats >/dev/null
done
Steve
On Mon, Sep 10, 2018 at 7:21 AM Steve Malenfant <smalenfant@gmail.com>
wrote:
> Ever since we upgraded to 6.x or 7.x, it seems like we have issues with
> memory utilization climbing over time in a 4GB VM. It takes in average
> about 5 days before it starts swapping.
>
> Current version: 7.1.4
>
> There is only a few requests getting to those VMs and they are creating
> quite a problem on the VM hosts due to excessive swapping (IO). Most of
> those requests, if not all are using the astats_over_http plugin for
> Traffic Control.
>
> I've condensed the memory dump just to show what's being used.
> Seems like hdrStrHeap seems to be using quite a bit of memory and keeps
> increasing over time.
>
> Is there anything specific I would need to look into?
>
>
> -----------------------------------------------------------------------------------------
> Allocated | In-Use | Type Size | Free List Name
>
> --------------------|--------------------|------------|----------------------------------
> 2097152 | 0 | 32768 |
> memory/ioBufAllocator[8]
> 524288 | 0 | 8192 |
> memory/ioBufAllocator[6]
> 96993280 | 96661504 | 4096 |
> memory/ioBufAllocator[5]
> 16384 | 128 | 128 |
> memory/ioBufAllocator[0]
> 73728 | 45312 | 96 |
> memory/eventAllocator
> 1407600 | 1399440 | 80 |
> memory/mutexAllocator
> 24576 | 22016 | 64 |
> memory/ioBlockAllocator
> 1150560 | 1145328 | 48 |
> memory/ioDataAllocator
> 65280 | 62400 | 240 | memory/ioAllocator
> 97760 | 25568 | 752 |
> memory/netVCAllocator
> 114400 | 0 | 880 |
> memory/sslNetVCAllocator
> 67712 | 33856 | 33856 |
> memory/dnsBufAllocator
> 163840 | 0 | 1280 |
> memory/dnsEntryAllocator
> 4096 | 16 | 16 |
> memory/expiryQueueEntry
> 8192 | 64 | 64 |
> memory/refCountCacheHashingValueAllocator
> 296960 | 2320 | 2320 |
> memory/hostDBContAllocator
> 1162870784 | 1162797056 | 2048 | memory/hdrStrHeap
> 1162870784 | 1162862592 | 2048 | memory/hdrHeap
> 24576 | 6144 | 192 |
> memory/httpServerSessionAllocator
> 1990656 | 0 | 7776 |
> memory/httpSMAllocator
> 114688 | 30464 | 896 |
> memory/http1ClientSessionAllocator
> 24576 | 6624 | 96 |
> memory/INKContAllocator
> 4096 | 224 | 32 |
> memory/apiHookAllocator
> 262144 | 0 | 1024 | memory/ArenaBlock
> 2431268112 | 2425101056 | | TOTAL
>
> -----------------------------------------------------------------------------------------
>
> Steve
>
|