I have just many questions.
Today, I use single process ManifoldCF. To crawl, I have 5 jobs by server and I have a script to check if last job is finish and start the following. I crawl just 1 job in the same time / server.
The multiprocess can be useful for a configuration like this?
And a second question:
What is the utility of Zookeeper for multiprocess? To distribute the configuration?
And the last:
Multiprocess ManifoldCF is stable for production env? And with Zk?