manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From LIROT Daniel - SG/SPSSI/CPII/DOSO/ET <Daniel.Li...@developpement-durable.gouv.fr>
Subject ManifoldCF + Postgresql - long freeze on job
Date Fri, 08 Feb 2019 13:07:37 GMT
Hello,

We use ManifoldCF v2.10, with postgresql (9.6) to crawl our websites.
this represents approximately 1.2 million documents.
We split the crawl into 4 jobs that distribute their results on 3 SOLR 
collections.
The crawl is powerful up to 500000 documents (25000 to 30000 docs / 
hour) then the performance decreases strongly in progress, we observe 
freezes very very long, you might think that the crawl is stopped.
We suspect a reindexing, noticeably of the intrinsiclink table which is 
very important 85 Million lines.
Is it possible to prohibit re-indexing controlled by manifoldCF?
An other idea ?

best Regards
LIROT daniel
-- 

Mime
View raw message