manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: max_pred_locks_per_transaction
Date Fri, 25 Jan 2013 11:24:42 GMT
Hi Erlend,

Leaving logging at the default values would have shown the ERROR
message you have below.  So the cause for the pause must have been
something else.

When ManifoldCF seems to make no progress, the first thing to do is
look at the simple history and see if it is retrying on something for
some reason.  If that is not helpful, get a thread dump.  You can use
jstack for that purpose.

As for the PostgreSQL parameters, max_pred_locks_per_transaction seems
to be PostgreSQL 9 black magic.  Here's the documentation:

http://www.postgresql.org/docs/9.1/static/runtime-config-locks.html

Default is 64, but they don't say how it is allocated.  I'd guess
therefore that you should try 50% higher and see if that works, e.g.
96.  I guess the limit is the amount of shared memory your OS allows
you to allocate.

Karl

On Fri, Jan 25, 2013 at 5:26 AM, Erlend GarĂ¥sen <e.f.garasen@usit.uio.no> wrote:
>
> After we started to crawl journals, the crawler just stopped after a couple
> of hours. Running MCF in debug mode gave me the stack trace shown below. I
> thing we need to adjust some PG parameters, perhaps
> max_pred_locks_per_transaction. The database admins are now asking me about
> which value to set. They have increased it, but I don't know whether it is
> sufficient.
>
> Another thing. I did not see this message until I changed the log level to
> debug, but the log4j should catch this error messages with warn level
> enabled. So maybe it is a dead end, i.e. a totally different cause and that
> this just occurred as a coincidence.
>
> ERROR 2013-01-19 03:47:15,049 (Worker thread '49') - Worker thread
> aborting and restarting due to database connection reset: Database
> exception: Exception doing query: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: Exception doing query: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:681)
> at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
> at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
> at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
> at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
> at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1863)
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:554)
> Caused by: org.postgresql.util.PSQLException: ERROR: out of shared memory
> Hint: You might need to increase max_pred_locks_per_transaction.
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
> at org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
> DEBUG 2013-01-19 03:47:23,386 (Idle cleanup thread) - Checking for
> connections, idleTimeout: 1358563583386
>
> --
> Erlend GarĂ¥sen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message