manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: MFCF UI Hangs
Date Tue, 18 Dec 2012 02:10:47 GMT
Hi Arcadius,

The best model for production deployment depends on what you are
trying to do, how large/many processors your machine has, etc.  The
machine's resources must be shared among the crawler, the UI, the API,
and usually the database.  For a crawl that is going flat-out, with
lots and lots of worker threads, therefore, the UI is going to get
relatively short shrift, and making sure that the UI has priority
requires not only changing how locks work in ManifoldCF but how they
work in the underlying database as well.  So this is a very
challenging problem for us to solve.

If this is very important to you, here are my recommendations:

(1) You probably want to start with a performance analysis.  It may
well be that the cause of the problem you are seeing is something like
JDBC connection pool starvation.  I say this because most people find
the UI slow but usable even when the crawler is pounding away.  Please
read the following document and make sure your database and ManifoldCF
parameters all make sense:
http://manifoldcf.apache.org/release/trunk/en_US/performance-tuning.html
.  If all seems well already, get a few thread dumps at the time when
the UI is unresponsive for you, post them here, and I can interpret
them to tell you what the cause of the delay actually is.

(2) For production, if you run the software on Linux and use a
multiprocess model, you will have a lot more control over what
processes have priority.  For instance, you can use the "nice" utility
to set priorities such that the UI web application has higher process
priority than the crawler.  Alternatively, we can experiment here with
setting lower priority for the crawler's worker threads; currently we
don't do any of this.  This will not help if there is a resource
starvation problem, so please look at (1) first.

(3) In my experience, the lock that is slow for the job status page is
the full table lock for the Jobs table.  This is a database lock which
is hard for the UI to get, but it is unnecessary to actually control a
job.  Instead it is needed to display job statistics.  So if you had
to, you could use the REST API to pause one or all of your jobs, after
which the UI should be very fast again until they are restarted.


Karl


On Mon, Dec 17, 2012 at 8:30 PM, Arcadius Ahouansou
<arcadius@menelic.com> wrote:
>
> Hello Karl.
>
> I am using the quick-start single process mode.
> Please, is this the recommended setting for production deployment?
>
> I found it unpleasant that the UI freezes and when it happens, I cannot
> pause the crawl as the UI  becomes unusable.
> It would be good to have a lock-free app that would read only the already
> committed data.
>
> Thanks.
>
> Arcadius.
>
>
> On 10 December 2012 08:29, Karl Wright <daddywri@gmail.com> wrote:
>>
>> Hi Arcadius,
>>
>> First, are you using ManifoldCF in single-process or multiprocess
>> mode?  If you are using single-process mode, then the agents process
>> and the web application both use the same process, so a busy crawl can
>> interfere enough with the web application so that bringing up the web
>> application becomes a challenge.
>>
>> Even in multiprocess mode, though, a busy crawl will often block the
>> web UI from obtaining some of the locks it needs to display certain
>> kinds of information.  The job status screen especially is affected by
>> this, since the screen requires global locks on the jobstatus table in
>> order to calculate the statistics it displays.  Changing how the locks
>> work might help this problem - but there is not yet a ticket for this
>> work.  You may also find that pausing any jobs that are causing
>> significant load will allow you to work with the UI, and you can
>> resume the jobs when you are done.
>>
>> Karl
>>
>> On Mon, Dec 10, 2012 at 3:17 AM, Arcadius Ahouansou
>> <arcadius@menelic.com> wrote:
>> >
>> > Hello.
>> >
>> > I started with Manifold CF5.1 and noticed that when the web crawler is
>> > running, at some point, the admin UI admin UI hangs.
>> >
>> > I thought that it may be due to the embedded java db.
>> > So upgraded to PostgreSQL and still seen similar problems.
>> >
>> > Now, I am running on MFCF1.1 on PostgreSQL 9.2 and still same issue.
>> >
>> > When a click on any of the links such as
>> > List Output Connections
>> > List Authority Connections
>> > List Repository Connections
>> > List all Jobs etc
>> >
>> > The UI takes a very long time and than I see nothing but the blue screen
>> > as
>> > shown at
>> > http://dl.dropbox.com/u/43641128/manifold.jpg
>> >
>> > Note that after I restart, things are OK for a while...
>> >
>> > This is on a fresh start of jetty, and the only one error in the log
>> > file is
>> > shown below.
>> >
>> > My Java version:
>> > java version "1.7.0_05"
>> > Java(TM) SE Runtime Environment (build 1.7.0_05-b06)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)
>> >
>> > I have also captured a thread dump that can be seen at
>> > http://dl.dropbox.com/u/43641128/stack.log
>> >
>> >
>> >
>> > <log>
>> > Starting jetty...
>> > 123 [main] INFO org.eclipse.jetty.server.Server - jetty-7.5.4.v20111024
>> > 183 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration - Extract
>> >
>> > jar:file:/home/arcad/apache-manifoldcf-1.0.1/web/war/mcf-crawler-ui.war!/ to
>> > /tmp/jetty-0.0.0.0-8345-mcf-crawler-ui.war-_mcf-crawler-ui-any-/webapp
>> > 1452 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> > started
>> >
>> > o.e.j.w.WebAppContext{/mcf-crawler-ui,file:/tmp/jetty-0.0.0.0-8345-mcf-crawler-ui.war-_mcf-crawler-ui-any-/webapp/},/home/arcad/apache-manifoldcf-1.0.1/example/./../web/war/mcf-crawler-ui.war
>> > 1528 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration - Extract
>> >
>> > jar:file:/home/arcad/apache-manifoldcf-1.0.1/web/war/mcf-authority-service.war!/
>> > to
>> >
>> > /tmp/jetty-0.0.0.0-8345-mcf-authority-service.war-_mcf-authority-service-any-/webapp
>> > 2349 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> > started
>> >
>> > o.e.j.w.WebAppContext{/mcf-authority-service,file:/tmp/jetty-0.0.0.0-8345-mcf-authority-service.war-_mcf-authority-service-any-/webapp/},/home/arcad/apache-manifoldcf-1.0.1/example/./../web/war/mcf-authority-service.war
>> > 2354 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration - Extract
>> >
>> > jar:file:/home/arcad/apache-manifoldcf-1.0.1/web/war/mcf-api-service.war!/
>> > to
>> > /tmp/jetty-0.0.0.0-8345-mcf-api-service.war-_mcf-api-service-any-/webapp
>> > 3079 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
>> > started
>> >
>> > o.e.j.w.WebAppContext{/mcf-api-service,file:/tmp/jetty-0.0.0.0-8345-mcf-api-service.war-_mcf-api-service-any-/webapp/},/home/arcad/apache-manifoldcf-1.0.1/example/./../web/war/mcf-api-service.war
>> > 3148 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started
>> > SelectChannelConnector@0.0.0.0:8345 STARTING
>> > Jetty started.
>> > Starting crawler...
>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Job
>> > 1352077408353
>> > is not active
>> >         at
>> > org.apache.manifoldcf.crawler.jobs.Jobs.abortJob(Jobs.java:1706)
>> >         at
>> >
>> > org.apache.manifoldcf.crawler.jobs.JobManager.manualAbort(JobManager.java:5436)
>> >         at org.apache.jsp.execute_jsp._jspService(execute_jsp.java:1377)
>> >         at
>> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >         at
>> >
>> > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:388)
>> >         at
>> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
>> >         at
>> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >         at
>> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:480)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
>> >         at
>> >
>> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)
>> >         at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:941)
>> >         at
>> >
>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)
>> >         at
>> >
>> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
>> >         at
>> >
>> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
>> >         at org.eclipse.jetty.server.Server.handle(Server.java:349)
>> >         at
>> >
>> > org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)
>> >         at
>> >
>> > org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:936)
>> >         at
>> > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)
>> >         at
>> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)
>> >         at
>> >
>> > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)
>> >         at
>> >
>> > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)
>> >         at
>> >
>> > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)
>> >         at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
>> >         at
>> >
>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
>> >         at java.lang.Thread.run(Thread.java:722)
>> >
>> > </log>
>> >
>> > Thanks.
>> >
>> > Arcadius
>> >
>
>
>
>

Mime
View raw message