manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
Date Mon, 06 Mar 2017 09:24:06 GMT
Hi Guy,

(1) I have no experience with PostgresSQL versions beyond 9.3, but I doubt
you would have problems.
(2) If you are using multiple processes, even if there's only one agents
process, you must use synchronization.  I would recommend Zookeeper;
file-system-based synchronization is deprecated.
(3) Windows has many ways of interfering with file-based sync, including
path-length issues.  I have seen Windows fail to unlock files and need a
reboot to release the lock.  This is one reason why file-system-based
locking is deprecated.

Thanks,
Karl


On Mon, Mar 6, 2017 at 4:03 AM, Standen Guy <Guy.Standen@uk.fujitsu.com>
wrote:

> Hi Karl,
>
>                 Thanks for that I will try version 2.6.  Whilst moving to
> MCF 2.6 I would potentially like to upgrade my backend PostgreSQL version
> from 9.3.5.
>
> 1) Do you have a recommendation for which PostgreSQL to use with MCF 2.6
>   e.g. PostgreSQL 9.3.16 or PostgreSQL 9.6.2?
>
> 2) For a production system on a single server running a single MCF agents
> process would you recommend the file based synchronisation locking or
> zookeeper based synchronisation locking.  With the file based
> synchronisation locking mechanism I have sometimes seen errors of the form :
>
>                 'D:\Apps\ManifoldCF\apache-manifoldcf-2.0.1\multiprocess-
> file-example\.\.\syncharea\475\708\lock-_POOLTARGET__OUTPUTCONNECTORPOOL_Solr
> COLL1 osp_unstruct.lock' failed: Access is denied’ ( I have ensured that
> the SYNCHAREA  is not  scanned by AV or Indexed by Windows Search  and all
> MCF processes run as the same user)
>
> What could cause these errors?
>
>
>
> Many Thanks,
>
>
>
> Guy
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* 03 March 2017 17:35
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
>
>
>
> Hi Guy:
>
>
>
> It is expected that sometimes database deadlock will develop, and the
> transaction will need to be retried.  There is code in MCF for doing this:
>
>
>
> >>>>>>
>
>     if (sqlState != null && sqlState.equals("40001"))
>
>       return new ManifoldCFException(message,e,
> ManifoldCFException.DATABASE_TRANSACTION_ABORT);
>
> <<<<<<
>
>
>
> I suspect that your version of MCF is old enough so that this particular
> error and the associated retry are not taking place.  Upgrading to 2.6 will
> definitely help there.
>
>
>
> Karl
>
>
>
>
>
> On Fri, Mar 3, 2017 at 11:17 AM, Standen Guy <Guy.Standen@uk.fujitsu.com>
> wrote:
>
> Hi Karl,
>
>                 Thanks for coming back so quickly. Unfortunately I wasn’t
> using a JCIFS connection. One of the issues I was seeing was between a
> crawl of an intranet site (no explicit throttling other than number of
> connections) and   scheduled crawl (every 5 mins) to a relational DB via
> JDBC connector again no explicit throttling.  To simplify things both jobs
> are using a NULL output connection. Sometimes both the Web crawl and the
> JDBC connection can run together but at other times 1 or both jobs will
> appear to lock up with just a few active documents showing. When I get a
> lock up the mcf log contains errors like:
>
>
>
> “DEBUG 2017-03-03 15:28:20,466 (Worker thread '5') - Exception Database
> exception: SQLException doing query (40001): ERROR: could not serialize
> access due to read/write dependencies among transactions”
>
>
>
> See the attached log extract for a little more detail.
>
>
>
> Any view why this might be happening?
>
>
>
> Best Regards,
>
>
>
> Guy
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* 03 March 2017 11:27
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
>
>
>
> Hi Guy,
>
>
>
> A issue with concurrent jobs is known for jobs sharing the same JCIFS
> connection.  Is that what you are using?  This has nothing to do with the
> version of Postgresql you are using; it has to do with what "bins"
> documents are thought to come from.  There has been a recent improvement
> for this issue, which will be released in April.  See
> https://issues.apache.org/jira/browse/CONNECTORS-1364.
>
>
>
> The current version of MCF (2.6) supports Solr 6.x.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Fri, Mar 3, 2017 at 5:27 AM, Standen Guy <Guy.Standen@uk.fujitsu.com>
> wrote:
>
> Hi Karl,
>
> I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and have
> had some issues with multiple jobs running concurrently.
>
> I am considering upgrading to MCF 2.6 and to a newer version of
> PostgreSQL. Would you be able to advise which version of PostgreSQL I
> should consider using with MCF 2.6 (e.g.  PostgreSQL  9.3.16 or 9.6.2)
>
>
>
> I am also considering upgrading from SOLR 4.10.3 to a newer version. The
> MCF compatibility matrix mentions that compatibility has been tested to
> SOLR version 4.5.1. Do you have any advice about compatibility with the
> newer versions of SOLR e.g. 6.4.1.
>
>
>
> Best Regards
>
>
>
> Guy
>
>
>
>
> Unless otherwise stated, this email has been sent from Fujitsu Services
> Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in
> England No 2216100) both with registered offices at: 22 Baker Street,
> London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and
> Fujitsu Laboratories of Europe Limited (registered in England No. 4153469)
> both with registered offices at: Hayes Park Central, Hayes End Road, Hayes,
> Middlesex, UB4 8FE.
> This email is only for the use of its intended recipient. Its contents are
> subject to a duty of confidence and may be privileged. Fujitsu does not
> guarantee that this email has not been intercepted and amended or that it
> is virus-free.
>
>
>
>
> Unless otherwise stated, this email has been sent from Fujitsu Services
> Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in
> England No 2216100) both with registered offices at: 22 Baker Street,
> London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and
> Fujitsu Laboratories of Europe Limited (registered in England No. 4153469)
> both with registered offices at: Hayes Park Central, Hayes End Road, Hayes,
> Middlesex, UB4 8FE.
> This email is only for the use of its intended recipient. Its contents are
> subject to a duty of confidence and may be privileged. Fujitsu does not
> guarantee that this email has not been intercepted and amended or that it
> is virus-free.
>
>
>
> Unless otherwise stated, this email has been sent from Fujitsu Services
> Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in
> England No 2216100) both with registered offices at: 22 Baker Street,
> London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and
> Fujitsu Laboratories of Europe Limited (registered in England No. 4153469)
> both with registered offices at: Hayes Park Central, Hayes End Road, Hayes,
> Middlesex, UB4 8FE.
> This email is only for the use of its intended recipient. Its contents are
> subject to a duty of confidence and may be privileged. Fujitsu does not
> guarantee that this email has not been intercepted and amended or that it
> is virus-free.
>

Mime
View raw message