httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Dunigan <>
Subject Re: [users@httpd] MSN-Bot doesn't finish a download?
Date Tue, 03 May 2005 14:22:29 GMT
On Tue, 3 May 2005, Joshua Slive wrote:

> On 5/3/05, Abu Hurayrah <> wrote:
> > I've tried sizes ranging from 50,000 bytes to 500,000 bytes, and always,
> > MSN gets only that much.
> It's very possible that the bot is interested in less than 50,000 bytes.
> > 
> > Previously, MSN would download the ENTIRE files, when I was sending
> > these files all at once.  I cannot understand the mechanism that
> > prevents it from continuing downloading the entire file, despite the
> > fact that I partition the download into these discrete chunks.  I am not
> > mangling the data in any way I know, I am simply sending it down in
> > chunks to reduce the memory footprint of each of my download script's
> > instances.
> It is possible that the bot was always dropping the connection in the
> same place, but you didn't know about it before because the rest of
> the content would just get blindly sent.
> Joshua.

Joshua's explanations seem the most likely, but I thought I'd toss out
that, in my experience, MS products are often intolerant of latency.  It
comes from a corporate culture that began when everything ran on a LAN.  
Perhaps the MSN bot just can't deal with the split-second pause between
chunks, assumes the connection went bad, and drops it.  Perhaps it's not
even a bug, and they deliberately built it that way as a safeguard against
remote site slowdowns.  At any rate, it is most certainly a client issue,
not an Apache issue.  If the dropped connections cause you trouble, you'll
probably have to practice some defensive programming and handle that agent
ID differently in your download script.

The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:> for more info.
To unsubscribe, e-mail:
   "   from the digest:
For additional commands, e-mail:

View raw message