manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmet Arslan <iori...@yahoo.com>
Subject hanging crawler
Date Mon, 24 Jun 2013 11:59:15 GMT
Hello All,

I hava a MCF 1.2 setup ( with postgresql-9.2) where I crawl some newspaper sites using Web
connectors.

I use following setting for jobs:

Maximum hop count for link type 'link': 1
Maximum hop count for link type 'redirect': Unlimited
Hop count mode: No deletes, forever

Start method:Start at beginning of schedule window
Schedule type:Scan every document once
Maximum run time: 90 minutes

I scheduled jobs to run every two hours. However after some crawl hangs. I found these exceptions
in the log.

What could be wrong? Any suggestions?

Thanks,
Ahmet

ERROR 2013-06-24 10:39:34,999 (Worker thread '1') - Worker thread aborting and restarting
due to database connection reset: Database exception: SQLException doing query (25P02): ERROR:
current transaction is aborted, commands ignored until end of transaction block
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database exception: SQLException
doing query (25P02): ERROR: current transaction is aborted, commands ignored until end of
transaction block
at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:717)
at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:745)
at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1430)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:822)
at org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4148)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:2017)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1948)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:562)
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands
ignored until end of transaction block
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:862)
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:677)
ERROR 2013-06-24 10:39:33,473 (Worker thread '1') - Explain failed with error Database exception:
SQLException doing query (40001): ERROR: could not serialize access due to read/write dependencies
among transactions
  Detail: Reason code: Canceled on identification as a pivot, during conflict out checking.
  Hint: The transaction might succeed if retried.
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database exception: SQLException
doing query (40001): ERROR: could not serialize access due to read/write dependencies among
transactions
  Detail: Reason code: Canceled on identification as a pivot, during conflict out checking.
  Hint: The transaction might succeed if retried.
at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:717)
at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:745)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.explainQuery(DBInterfacePostgreSQL.java:1233)
at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1449)
at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:822)
at org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4148)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:2017)
at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.flush(WorkerThread.java:1948)
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:562)
Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to read/write
dependencies among transactions
  Detail: Reason code: Canceled on identification as a pivot, during conflict out checking.
  Hint: The transaction might succeed if retried.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
at org.apache.manifoldcf.core.database.Database.execute(Database.java:862)
at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:677)
Mime
View raw message