manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Standen Guy <Guy.Stan...@uk.fujitsu.com>
Subject RE: Advice on which PostgreSQL to use with ManifoldCF 2.6
Date Mon, 06 Mar 2017 09:03:45 GMT
Hi Karl,
                Thanks for that I will try version 2.6.  Whilst moving to MCF 2.6 I would
potentially like to upgrade my backend PostgreSQL version from 9.3.5.
1) Do you have a recommendation for which PostgreSQL to use with MCF 2.6   e.g. PostgreSQL
9.3.16 or PostgreSQL 9.6.2?
2) For a production system on a single server running a single MCF agents process would you
recommend the file based synchronisation locking or zookeeper based synchronisation locking.
 With the file based synchronisation locking mechanism I have sometimes seen errors of the
form :
                'D:\Apps\ManifoldCF\apache-manifoldcf-2.0.1\multiprocess-file-example\.\.\syncharea\475\708\lock-_POOLTARGET__OUTPUTCONNECTORPOOL_Solr
COLL1 osp_unstruct.lock' failed: Access is denied’ ( I have ensured that the SYNCHAREA 
is not  scanned by AV or Indexed by Windows Search  and all MCF processes run as the same
user)
What could cause these errors?

Many Thanks,

Guy

From: Karl Wright [mailto:daddywri@gmail.com]
Sent: 03 March 2017 17:35
To: user@manifoldcf.apache.org
Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6

Hi Guy:

It is expected that sometimes database deadlock will develop, and the transaction will need
to be retried.  There is code in MCF for doing this:

>>>>>>
    if (sqlState != null && sqlState.equals("40001"))
      return new ManifoldCFException(message,e,ManifoldCFException.DATABASE_TRANSACTION_ABORT);
<<<<<<

I suspect that your version of MCF is old enough so that this particular error and the associated
retry are not taking place.  Upgrading to 2.6 will definitely help there.

Karl


On Fri, Mar 3, 2017 at 11:17 AM, Standen Guy <Guy.Standen@uk.fujitsu.com<mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
Hi Karl,
                Thanks for coming back so quickly. Unfortunately I wasn’t using a JCIFS
connection. One of the issues I was seeing was between a crawl of an intranet site (no explicit
throttling other than number of connections) and   scheduled crawl (every 5 mins) to a relational
DB via JDBC connector again no explicit throttling.  To simplify things both jobs  are using
a NULL output connection. Sometimes both the Web crawl and the JDBC connection can run together
but at other times 1 or both jobs will appear to lock up with just a few active documents
showing. When I get a lock up the mcf log contains errors like:

“DEBUG 2017-03-03 15:28:20,466 (Worker thread '5') - Exception Database exception: SQLException
doing query (40001): ERROR: could not serialize access due to read/write dependencies among
transactions”

See the attached log extract for a little more detail.

Any view why this might be happening?

Best Regards,

Guy






From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: 03 March 2017 11:27
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6

Hi Guy,

A issue with concurrent jobs is known for jobs sharing the same JCIFS connection.  Is that
what you are using?  This has nothing to do with the version of Postgresql you are using;
it has to do with what "bins" documents are thought to come from.  There has been a recent
improvement for this issue, which will be released in April.  See https://issues.apache.org/jira/browse/CONNECTORS-1364.

The current version of MCF (2.6) supports Solr 6.x.

Thanks,
Karl


On Fri, Mar 3, 2017 at 5:27 AM, Standen Guy <Guy.Standen@uk.fujitsu.com<mailto:Guy.Standen@uk.fujitsu.com>>
wrote:
Hi Karl,
I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and have had some issues with
multiple jobs running concurrently.
I am considering upgrading to MCF 2.6 and to a newer version of PostgreSQL. Would you be able
to advise which version of PostgreSQL I should consider using with MCF 2.6 (e.g.  PostgreSQL
 9.3.16 or 9.6.2)

I am also considering upgrading from SOLR 4.10.3 to a newer version. The MCF compatibility
matrix mentions that compatibility has been tested to SOLR version 4.5.1. Do you have any
advice about compatibility with the newer versions of SOLR e.g. 6.4.1.

Best Regards

Guy


Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty
of confidence and may be privileged. Fujitsu does not guarantee that this email has not been
intercepted and amended or that it is virus-free.


Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty
of confidence and may be privileged. Fujitsu does not guarantee that this email has not been
intercepted and amended or that it is virus-free.


Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered
in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered
offices at: 22 Baker Street, London W1U 3BW;  PFU (EMEA) Limited, (registered in England No
1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both
with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.

This email is only for the use of its intended recipient. Its contents are subject to a duty
of confidence and may be privileged. Fujitsu does not guarantee that this email has not been
intercepted and amended or that it is virus-free.
Mime
View raw message