From users-return-319-apmail-trafficserver-users-archive=trafficserver.apache.org@trafficserver.apache.org Fri Sep 17 02:58:29 2010 Return-Path: Delivered-To: apmail-trafficserver-users-archive@www.apache.org Received: (qmail 58869 invoked from network); 17 Sep 2010 02:58:29 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 17 Sep 2010 02:58:29 -0000 Received: (qmail 66680 invoked by uid 500); 17 Sep 2010 02:58:29 -0000 Delivered-To: apmail-trafficserver-users-archive@trafficserver.apache.org Received: (qmail 66541 invoked by uid 500); 17 Sep 2010 02:58:28 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 66533 invoked by uid 99); 17 Sep 2010 02:58:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 02:58:27 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [71.6.165.248] (HELO kramer.ogre.com) (71.6.165.248) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 02:58:19 +0000 Received: from loki.ogre.com (homey.ogre.com [24.56.189.145]) (authenticated bits=0) by kramer.ogre.com (8.14.3/8.14.3) with ESMTP id o8H2vvZ5023167 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 16 Sep 2010 19:57:57 -0700 X-DKIM: Sendmail DKIM Filter v2.8.3 kramer.ogre.com o8H2vvZ5023167 Message-ID: <4C92D935.3000406@apache.org> Date: Thu, 16 Sep 2010 20:57:57 -0600 From: Leif Hedstrom User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9) Gecko/20100907 Fedora/3.1.3-1.fc13 Thunderbird/3.1.3 MIME-Version: 1.0 To: Pranav Desai CC: users@trafficserver.apache.org Subject: Re: Need some help tuning for large file performance References: <4C92C6B5.1070909@apache.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 09/16/2010 08:09 PM, Pranav Desai wrote: > On > Done. Using 2 disks. Do you recommend a raid config for better performance ? No, I recommend RAID only if you really, really care for redundancy / reliability. ATS will automatically "load balance" across any disks that you throw at it. > > Here are my tcp mem parameters. And req/sec isnt of concern here so I > should be ok with the listen queue and backlogs. If you have any > particular setting in mind please let me know. > > net.ipv4.tcp_mem = 1339776 1786368 2679552 > net.ipv4.tcp_wmem = 4096 87380 8388608 > net.ipv4.tcp_rmem = 4096 87380 8388608 These would only make a difference if there's a latency between the client and the server (which is generally not the case in a "lab"). The above are for the autoscaling window sizes I think, which seem reasonable. There are settings in records.config to bump up the initial window size for the connections: CONFIG proxy.config.net.sock_send_buffer_size_in INT 262144 CONFIG proxy.config.net.sock_recv_buffer_size_in INT 0 CONFIG proxy.config.net.sock_send_buffer_size_out INT 0 CONFIG proxy.config.net.sock_recv_buffer_size_out INT 0 You definitely want to tune that mmap setting I mentioned earlier, and a few other interesting sysctl's would be net.ipv4.tcp_max_syn_backlog (set it pretty high) net.core.somaxconn net.ipv4.tcp_syncookies (enable it) net.ipv4.ip_local_port_range net.ipv4.tcp_ecn (probably want it disabled) net.ipv4.tcp_max_tw_buckets (increase for lots of sockets I think) > > I think I can reproduce it but under load, so it might be a bit > difficult to debug it especially with all the threads. I will try to > get to a simpler test case to reproduce it. Maybe I can run > traffic_server alone with a single network and io thread ? How do you > guys normally debug it ? I think we should move the discussions related to this crasher problem to the dev@trafficserver.apache.org mailing list (information how to subscribe to it is on http://trafficserver.apache.org). There's a wider crowd there that might be able to help as well, in particular John Plevyak knows the cache better than anyone else on the planet. That much said, if you can reproduce it with restrictions like you mention, that'll certainly help. Or, just describe how to setup the environment, and what "load" to send to it, that might also help. But the more you can limit the parameters / tests / time necessary to reproduce it, the better. Thanks! -- leif