As it says on the tin, our connections via svn+ssh are painfully slow, yet we can ssh into the server without any delays whatsoever.  A find on the subversion repository likewise whips through without delay, and there's no memory or CPU load on the server when people are connecting.  So all signs point to some sort of connection or authentication delay.


I ran ssh through strace and compared that to a svn connection through strace.  Here's where things go awry.  First, SSH which works:


SSH:


socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("{{Our IP}}")}, 16) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("{{Our IP}}")}, [128->16]) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("{{Our IP}}")}, [128->16]) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("{{Our IP}}")}, [128->16]) = 0
stat("/home/fdps/.ssh", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 4


... and so forth; it keeps on going normally.  But with svn+ssh:


socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("{{Our IP}}")}, 16) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len=26}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CTRUNC|MSG_DONTWAIT|MSG_EOR|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba0010}, msg_len=26}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", iov_len=26}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_OOB|MSG_PEEK|MSG_DONTROUTE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_BATCH|MSG_FASTOPEN|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=26}], 2, MSG_NOSIGNAL) = 2
poll([{fd=3, events=POLLIN}], 1, 5000)  = 0 (Timeout)
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 6
connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("{{Our IP}}")}, 16) = 0
poll([{fd=6, events=POLLOUT}], 1, 0)    = 1 ([{fd=6, revents=POLLOUT}])
sendmmsg(6, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len=26}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CTRUNC|MSG_DONTWAIT|MSG_EOR|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba0010}, msg_len=26}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", iov_len=26}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_OOB|MSG_PEEK|MSG_DONTROUTE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_BATCH|MSG_FASTOPEN|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=26}], 2, MSG_NOSIGNAL) = 2


... and so on, stuck in repeated timeouts until it gives up.  


Now, note the difference on the socket call.  Ssh uses a socket call with SOCK_STREAM / TCP, while svn+ssh uses SOCK_DGRAM|SOCK_NONBLOCK / IP.  Why would it do that?  That doesn't make any sense to me.  How can I change this?  


For a list of all socket() calls from each:


SSH:


socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
socket(AF_UNIX, SOCK_STREAM, 0)         = 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4


SVN+SSH:


socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 6
socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC|SOCK_NONBLOCK, NETLINK_ROUTE) = 3
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3


 - kv, Karen