subversion-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bai Wei <baiwei...@gmail.com>
Subject Re: Extremely slow checkout on a large repository
Date Sun, 14 Oct 2012 01:06:43 GMT
Hi Andy, Thank you for your prompt reply.

I found the performance problem was caused by 21406 files inside a single
folder, total size of this folder
is 255M. (sorry for this bad layout, but this is not in my control)

I've done a little more test over this folder today, and found if we
launch svnserve with "--memory-cache-size=8192 --cache-txdeltas yes
--cache-fulltexts yes", the performance is much better.

Here is my test environment:
Server side:
svn version 1.7.7 (r1393599)
model name      : Intel(R) Xeon(R) CPU           L5410  @ 2.33GHz
/dev/sdb on /export type xfs (rw)
             total       used       free     shared    buffers     cached
Mem:      16467548   15698252     769296          0        256   13319084

Client side:
svn version 1.6.11 (r934486)
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
tmpfs on /root/tmp type tmpfs (rw,size=4g)
             total       used       free     shared    buffers     cached
Mem:      12297388    9509928    2787460          0     663260    7344808

svn server is running on a SSD disk, and I'm checking out to tmpfs,
free memory is enough, so we are CPU-bound in these tests.

Case 1:
If we launch svnserve with default option, it takes 23 mins to checkout
this folder:
/usr/bin/time svn co svn://10.68.xx.xx/data/task
Checked out revision 89151.
185.41user 261.34system 23:00.37elapsed 32%CPU (0avgtext+0avgdata
449760maxresident)k
0inputs+0outputs (0major+28236minor)pagefaults 0swaps

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND


29149 root      20   0  162m  38m 1292 R 100.0 0.2   0:24.98 svnserve -d -r
/export/svn/repositories/

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND


21920 root      20   0  206m 7348 2908 S  3.4  0.1   0:01.46 svn


In this case bottleneck is server CPU.

Case 2:
With cache options, it takes only 8 mins.
/usr/bin/time svn co svn://10.68.xx.xx/data/task
Checked out revision 89151.
181.65user 253.39system 8:07.51elapsed 89%CPU (0avgtext+0avgdata
449744maxresident)k
0inputs+0outputs (0major+28235minor)pagefaults 0swaps

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29128 root      20   0 8312m 548m 1300 S  0.7  3.4   0:02.07 svnserve -d -r
/export/svn/repositories/ --memory-cache-size=8192 --cache-txdeltas yes
--cache-fulltexts yes

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND


21793 root      20   0  268m  59m 2908 R 89.1  0.5   3:36.61 svn

CPU usage on server side decreased dramatically when cache options is on,
memory cost is acceptable in my case
, Client CPU slowdown the checkout process this time.

I also tested to checkout this folder on server side machine, with svn
1.7.7, to tmpfs,
it takes only 16 seconds, this significant improvement must be attribute to
 new working copy library (WC-NG) in svn 1.7?
Thank SVN dev team for your continuous efforts in optimization.

BTW. it's still painfully slow to checkout this repository to hard disk,
if disk IO will be optimized in future version of svn , it will be a good
news.

2012/10/12 Andy Levy <andy.levy@gmail.com>

> On Fri, Oct 12, 2012 at 11:06 AM, Wei Bai <baiwei.cn@gmail.com> wrote:
> > Hi, Thanks for replying.
> >
> >
> >
> > I'm using svnserve 1.7.5 on a Dell R610 server ( Xeon E5620*2/16GB)
> >
> > The server is running CentOS 5.5, SSD disk is used to increase IO
> > performance.
> >
> > The repository is very large: 100K+ files, 100K+ revisions, total size of
> > the work copy is about 1.5G.
> >
> >
> >
> > When I what to check out a new work copy on another linux machine, I
> found
> > it's painfully
> >
> > slow, it takes about 2 hours. Can anybody tell me if this speed is
> normal?
> >
> > I noticed when the svn checkout command is running on client side, there
> is
> > a svnserve process with 100% CPU usage on server side,
> >
> > does this means the concurrent performance of svn is very bad for
> checkout
> > command?
>
> Not necessarily. In my environment, I typically hit I/O constraints
> limiting my checkout performance long before I hit CPU constraints.
>
> If you're performing a checkout and seeing 100% CPU utilization on the
> server, you're CPU-bound. With a faster processor, you may see
> improved checkout performance. With an SSD (assuming the repository is
> on the SSD), you're probably not I/O bound.
>
> How's your memory utilization while checking out? Is it possible that
> you're memory constrained and swapping out to disk?
>
> Because of how Subversion stores revisions, it must look at past
> revisions to construct the revision you're requesting. Depending upon
> a number of factors, this may become CPU and/or memory bound. It's a
> trade-off of performance vs. storage efficiency, and there is no one
> optimal setting for everyone's repository. The developers have
> selected a value which works well enough for most people.
>
> > And this svnserve process will not disappear immediately if I kill the
> > client side svn process, it will run for a long time with 100% CPU usage,
> > this might be a problem?
>
> It depends on how you're killing the client. If the server doesn't
> realize that the client has terminated, it'll keep trying to perform
> the checkout.
>
> > Could anyone give some advice on how to optimize the performance of svn
> on a
> > large repository?
>
> First you need to determine your limiting factor. In this case, it
> looks like CPU (or memory, once you look into that).
>

Mime
View raw message