manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmet Arslan <>
Subject Re: Update on "Two simultaneous web crawls hang"
Date Wed, 30 Apr 2014 20:56:56 GMT
Hi Tom,

Vote has just ended. Karl is now pushing artifacts.


On Wednesday, April 30, 2014 11:08 PM, Tom Rees <> wrote:

Thank you for your help. I would like to upgrade to ManifoldCF 1.6. Do you know when this
will be available?


On Fri, Apr 25, 2014 at 3:14 PM, Karl Wright <> wrote:

Hi Tom,
>If this was manifoldcf 1.5, then your problem is almost certainly the bandwidth throttling. 
The code has a bad bug that enforces throttling that is 1000 times too slow.  Either adjust
the throttle, or apply the patch, or upgrade to mcf 1.6.
>Sent from my Windows Phone
>From: Tom Rees
>Sent: 4/25/2014 2:23 PM
>To: user; Vasant Kumar; Steven Bennett; Amy Jocefczyk-Papa
>Subject: Update on "Two simultaneous web crawls hang"
>In an email on April 4, 2014 I reported that whenever I run two simultaneous web crawls
in ManifoldCF that both crawls will simply stop progressing after a short period of time.
When I look at the thread stack traces I see that there are many fetcher threads that are
waiting forever in wait() calls.
>I am still having that same problem, but I have tried different configurations. First,
in the previous email the tests used Postgres 9.3.2. I tried using Postgres 9.1.0, but the
web crawls hang in the same way. However, it seems like the web crawls took slightly longer
to stop making progress. Also, I have tried crawling different web sites, and I also tried
using a custom output connector that only saves the downloaded files to the local file system.
The problem persists. Also, another developer at our site has this same problem whenever he
runs multiple web crawls. Is this a known issue with the web crawler?
>Tom Rees
View raw message