We use ManifoldCF v2.10, with postgresql (9.6) to
crawl our websites.
this represents approximately 1.2 million
We split the crawl into 4 jobs that distribute
their results on 3 SOLR collections.
The crawl is powerful up to 500000 documents (25000
to 30000 docs / hour) then the performance decreases strongly in
progress, we observe freezes very very long, you might think
that the crawl is stopped.
We suspect a reindexing, noticeably of the
intrinsiclink table which is very important 85 Million lines.
Is it possible to prohibit re-indexing controlled
An other idea ?