manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bisonti Mario <Mario.Biso...@vimar.com>
Subject R: Error Job stop after repeatidly interruption
Date Tue, 27 Nov 2018 17:10:48 GMT
I don’t understand how to stop cleanly zookeper.
I start it by crontab at reboot:
@reboot /opt/manifoldcf/multiprocess-zk-example-proprietary/runzookeeper.sh

But now I tried to abort the job buti t hangs in state “Aborting”

When I didn’t use zookeper I used the lock-clean.sh script but in the multiprocess-zk-example-proprietary it doesn’t exist.
So I am in an hang situation…
I am not able how to close zookeeper and I am not able to abort job ☹
☹



Da: Karl Wright <daddywri@gmail.com>
Inviato: martedì 27 novembre 2018 16:58
A: user@manifoldcf.apache.org
Oggetto: Re: Error Job stop after repeatidly interruption

Hi Mario,

You can shut down ManifoldCF at any time and it will pick up where it left off.
But you should never shut down the database or recycle zookeeper while ManifoldCF is running, or you can expect it to lock up.  If you get into that state, you have to shut down both Zookeeper *and* ManifoldCF, then start Zookeeper, then start ManifoldCF.  It will then pick up where it left off.

Karl


On Tue, Nov 27, 2018 at 4:19 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Hallo.
No, I didn’t shut MCF during running job

My zookeeper.cfg is:
tickTime=5000
dataDir=zookeeper
clientPort=8349
maxClientCnxns=1000
autopurge.snapRetainCount=3
autopurge.purgeInterval=6

I tried to increase maxClientCnxns to 100000



Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: martedì 27 novembre 2018 09:20
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

Hi Mario,

This error:

>>>>>>
FATAL 2018-11-26T18:13:50,868 (Agents idle cleanup thread) - Error tossed: Can't release lock we don't hold
java.lang.IllegalStateException: Can't release lock we don't hold
<<<<<<

... means that Zookeeper failed in some way.  Did you shut it down while MCF was still running?

You should probably shut everything down and start it up again, including Zookeeper.  There are also some configuration properties Zookeeper has (see zookeeper.cfg) that you may need to modify to handle a very large crawl, so I'd read the "how to build and deploy" page to make sure you've got the parameters consistent with that.

Karl


On Tue, Nov 27, 2018 at 3:13 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Hallo Karl.
I enlarge the timeout to process the documents on Tika server and the job didn’t stop about repeatidly interruption but now the job is running but hangs.

In manifoldcf.log I see:
               at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-26T18:13:39,050 (Worker thread '63') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
               at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
               at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
               at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
FATAL 2018-11-26T18:13:50,868 (Agents idle cleanup thread) - Error tossed: Can't release lock we don't hold
java.lang.IllegalStateException: Can't release lock we don't hold
               at org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.releaseLock(ZooKeeperConnection.java:815) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearLock(ZooKeeperLockObject.java:218) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.clearGlobalWriteLockNoWait(ZooKeeperLockObject.java:100) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.LockObject.clearGlobalWriteLock(LockObject.java:160) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.LockObject.leaveWriteLock(LockObject.java:141) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.LockGate.leaveWriteLock(LockGate.java:205) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWrite(BaseLockManager.java:1224) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.core.lockmanager.BaseLockManager.leaveWriteLock(BaseLockManager.java:771) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.connectorcommon.throttler.ThrottleBin.poll(ThrottleBin.java:391) ~[?:?]
               at org.apache.manifoldcf.connectorcommon.throttler.Throttler$ThrottlingGroup.poll(Throttler.java:952) ~[?:?]
               at org.apache.manifoldcf.connectorcommon.throttler.Throttler$ThrottlingGroups.poll(Throttler.java:290) ~[?:?]
               at org.apache.manifoldcf.connectorcommon.throttler.Throttler.poll(Throttler.java:165) ~[?:?]
               at org.apache.manifoldcf.connectorcommon.throttler.ThrottleGroups.poll(ThrottleGroups.java:114) ~[?:?]
               at org.apache.manifoldcf.connectorcommon.interfaces.ThrottleGroupsFactory$ThrottlerPoll.doPoll(ThrottleGroupsFactory.java:65) ~[?:?]
               at org.apache.manifoldcf.core.system.ManifoldCF.pollAll(ManifoldCF.java:1491) ~[mcf-core.jar:?]
               at org.apache.manifoldcf.agents.system.IdleCleanupThread.run(IdleCleanupThread.java:93) [mcf-agents.jar:?]


It seems that the agent crashed?

What could I do?
I start the agent at the startup of the machine with the command, scheduled on crontab, by:
@reboot /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh

Thanks a lot Karl!



Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: lunedì 26 novembre 2018 21:51
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

Hi Mario,

As you can see, ManifoldCF retries some error conditions until it either succeeds or until it must finally give up.  At that point it can choose to abort the job or skip the document, depending on the error condition.

The reason I said to look for worker thread 87 in the first log was because I wanted to see if retried after that, or not.  The change I committed should make it retry for a period of time (10 minutes) before giving up.  The snippet of log you initially included showed a number of threads having difficulty connecting to the Tika service; that should be retried.  And your log messages show a ServiceInterruption was indeed processed, which will cause a retry to occur until the time interval has elapsed:

>>>>>>
WARN 2018-11-26T13:18:25,913 (Worker thread '88') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,039 (Worker thread '39') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,179 (Worker thread '8') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,232 (Worker thread '86') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,329 (Worker thread '87') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,668 (Worker thread '92') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,722 (Worker thread '99') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,862 (Worker thread '75') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,862 (Worker thread '12') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135%2F&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0E1iv%2FUpA5ARjT8yyrPJrQgFWZcEbePTD1yHDCOG4bU%3D&reserved=0>] failed: Connection refused (Connection refused)
<<<<<<

When a document causes Tika to crash, always, eventually ManifoldCF will give up and shut the job down, even though it's retried a number of times.  I warned you about this earlier; that may be what is happening here.  It would be good if you could verify that.

Karl


On Mon, Nov 26, 2018 at 12:19 PM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
After the error, the job stop.

After, I start the job and I obtain many times:

WARN 2018-11-26T16:32:55,003 (Worker thread '87') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
.
.
.
WARN 2018-11-26T17:18:34,709 (Worker thread '87') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
.
.
WARN 2018-11-26T17:18:34,709 (Worker thread '87') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
.
.

And teh job currently is running

Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: lunedì 26 novembre 2018 17:41
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

For any one of these warnings, e.g.:

WARN 2018-11-26T13:18:26,329 (Worker thread '87') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998

... look in the log for the next log message for "Worker thread '87'".  See what it did next and when.

Karl


On Mon, Nov 26, 2018 at 10:41 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Hallo Karl.
I built and I am using your trunk.
Thanks a lot.

But I have always the error on the job.

In the end of the manifoldcf.log I read:
               at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
               at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
               at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
               at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6]
               at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
               at org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor.addOrReplaceDocumentWithException(TikaExtractor.java:608) ~[?:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3226) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3077) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$MonitoredAddActivityWrapper.sendDocument(IncrementalIngester.java:3471) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.transformation.documentfilter.DocumentFilter.addOrReplaceDocumentWithException(DocumentFilter.java:208) ~[?:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3226) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3077) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2708) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756) ~[mcf-agents.jar:?]
               at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583) ~[mcf-pull-agent.jar:?]
               at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548) ~[mcf-pull-agent.jar:?]
               at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:939) ~[?:?]
               at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) ~[mcf-pull-agent.jar:?]
WARN 2018-11-26T13:18:25,913 (Worker thread '88') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,039 (Worker thread '39') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,179 (Worker thread '8') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,232 (Worker thread '86') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,329 (Worker thread '87') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,668 (Worker thread '92') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,722 (Worker thread '99') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,862 (Worker thread '75') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)
WARN 2018-11-26T13:18:26,862 (Worker thread '12') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to sengvivv01.local.domain:9998 [sengvivv01.local.domain/172.16.1.135<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.16.1.135&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=B7iOdxvrSA5KfIIIbyHpZgGQizMhwBFrLJarD5pD4Gw%3D&reserved=0>] failed: Connection refused (Connection refused)


So, I don’t understand if the worker tried to reconnect after 10 seconds or not

How could I check it?

Thanks a lot

Mario




Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: giovedì 15 novembre 2018 13:00
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

The easiest way to do it is just to check out current trunk:

svn co https://svn.apache.org/repos/asf/manifoldcf/trunk<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsvn.apache.org%2Frepos%2Fasf%2Fmanifoldcf%2Ftrunk&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=viJyEgDhFnQssTakG%2BeIn1LnPZhjlBwdmYUJTzyq5Po%3D&reserved=0>

No need to apply a patch then.  Just build:

ant make-core-deps
ant make-deps
ant build

Karl


On Thu, Nov 15, 2018 at 4:30 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Thanks a lot Karl.

To overwrite the connector with your patch, have I to download the trunk and recompile, isn’t it?

Excuse me for my questions but I am not expert on programming, compiling, etc.




Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: giovedì 15 novembre 2018 09:48
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

(1) I increased the retries to go at least 10 minutes.
(2) I handled the 503 response explicitly, with the same logic.

See: https://issues.apache.org/jira/browse/CONNECTORS-1556<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1556&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=xEnK14TELE6UAyTHM39b1vriO6RU%2BPX3ule%2BFcWBXUU%3D&reserved=0>

Karl


On Thu, Nov 15, 2018 at 3:35 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Yes, Karl.

Is it possible to apply the same your concept , wait 10 sec and retry three times , to the 503 error , too?

So, I would like to try, if, with the modification, I obtain that job end correctly instead of failure.


Thanks a lot
Mario

Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: giovedì 15 novembre 2018 09:17
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

Hi Mario,

Here's the code:

>>>>>>
            try {
              //System.out.println("About to do a content PUT");
              response = this.httpClient.execute(tikaHost, httpPut);
              //System.out.println("... content PUT succeeded");
            } catch (IOException e) {
              // Retry 3 times, 10000 ms between retries, and abort if doesn't work
              final long currentTime = System.currentTimeMillis();
              throw new ServiceInterruption("Tika down, retrying: "+e.getMessage(),e,currentTime + 10000L,
                -1L,3,true);
            }

            responseCode = response.getStatusLine().getStatusCode();
            if (response.getStatusLine().getStatusCode() == 200 || response.getStatusLine().getStatusCode() == 204) {
              tikaServerIs = response.getEntity().getContent();
              try {
                responseDs = new FileDestinationStorage();
                final OutputStream os2 = responseDs.getOutputStream();
                try {
                  IOUtils.copyLarge(tikaServerIs, os2, 0L, sp.writeLimit);
                } finally {
                  os2.close();
                }
                length = new Long(responseDs.getBinaryLength());
              } finally {
                tikaServerIs.close();
              }
            } else {
              activities.noDocument();
              if (responseCode == 422) {
                resultCode = "TIKASERVERREJECTS";
                description = "Tika Server rejected document with the following reason: "
                    + response.getStatusLine().getReasonPhrase();
                return handleTikaServerRejects(description);
              } else {
                resultCode = "TIKASERVERERROR";
                description = "Tika Server failed to parse document with the following error: "
                    + response.getStatusLine().getReasonPhrase();
                return handleTikaServerError(description);
              }
            }

          } catch (IOException | ParseException e) {
            resultCode = "TIKASERVERRESPONSEISSUE";
            description = e.getMessage();
            int rval;
            if (e instanceof IOException) {
              rval = handleTikaServerException((IOException) e);
            } else {
              rval = handleTikaServerException((ParseException) e);
            }
            if (rval == DOCUMENTSTATUS_REJECTED) {
              activities.noDocument();
            }
            return rval;
          }
<<<<<<
and
>>>>>>
  protected static int handleTikaServerError(String description)
      throws IOException, ManifoldCFException, ServiceInterruption {
    // MHL - what does Tika throw if it gets an IOException reading the stream??
    Logging.ingest.warn("Tika Server: Tika Server error: " + description);
    return DOCUMENTSTATUS_REJECTED;
  }
<<<<<<

The summary:

(1) If ManifoldCF cannot connect at all, or gets an IO error, it will wait at least 10 seconds and then retry -- up to three times.
(2) When Manifold sees a 503 error it immediately just rejects the document.

So you are requesting different handling for 503 errors?

Karl


On Thu, Nov 15, 2018 at 2:42 AM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Hallo Karl.
I opened an issue on Tika here:
https://issues.apache.org/jira/browse/TIKA-2776<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FTIKA-2776&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=TcHMlNFv%2B4xbwcay8KPNMfXyMJZSUR%2BQdABXWZjji9o%3D&reserved=0>

The person that develops tika, suggests me to put a waiting on the client (in my case manifoldcf)
https://issues.apache.org/jira/browse/TIKA-2776?focusedCommentId=16686620&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16686620<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FTIKA-2776%3FfocusedCommentId%3D16686620%26page%3Dcom.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel%23comment-16686620&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=6Z3xLrfu077R24HERFN6XxuQCFGzWrJbEZdruliwoC4%3D&reserved=0>

I am not able to do this…
Is it possible to implement on the MCF source?


Thanks a lot

Mario

Da: Karl Wright <daddywri@gmail.com<mailto:daddywri@gmail.com>>
Inviato: giovedì 8 novembre 2018 21:03
A: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Oggetto: Re: Error Job stop after repeatidly interruption

Hi Mario,

The Tika external connector retries for a while before it gives up and aborts the job.  If you can get the Tika server back up within a reasonable period of time all should be well.  But if one specific document *always* brings down the Tika server, it will be hard to recover from that.

Karl


On Thu, Nov 8, 2018 at 2:56 PM Bisonti Mario <Mario.Bisonti@vimar.com<mailto:Mario.Bisonti@vimar.com>> wrote:
Hallo.

I am trying to index more than 5000000 documents in a Windows Share.

It happens that job is interrupted due to repeatidly interruption.
This is the manifold.log:
.
.
WARN 2018-11-07T21:53:25,296 (Worker thread '59') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:53:25,476 (Worker thread '89') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:53:33,814 (Worker thread '15') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
        at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-07T21:53:57,861 (Worker thread '12') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
        at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
        at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-07T21:56:04,765 (Worker thread '24') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: The target server failed to respond
WARN 2018-11-07T21:56:04,770 (Worker thread '4') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: The target server failed to respond
WARN 2018-11-07T21:56:04,775 (Worker thread '95') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: The target server failed to respond
WARN 2018-11-07T21:56:04,793 (Worker thread '54') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: The target server failed to respond
ERROR 2018-11-07T21:56:04,802 (Worker thread '24') - Exception tossed: Repeated service interruptions - failure processing document: The target server failed to respond
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Repeated service interruptions - failure processing document: The target server failed to respond
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:489) [mcf-pull-agent.jar:?]
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor.addOrReplaceDocumentWithException(TikaExtractor.java:608) ~[?:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3226) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3077) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$MonitoredAddActivityWrapper.sendDocument(IncrementalIngester.java:3471) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.transformation.documentfilter.DocumentFilter.addOrReplaceDocumentWithException(DocumentFilter.java:208) ~[?:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3226) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3077) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2708) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756) ~[mcf-agents.jar:?]
        at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583) ~[mcf-pull-agent.jar:?]
        at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548) ~[mcf-pull-agent.jar:?]
        at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:939) ~[?:?]
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) ~[mcf-pull-agent.jar:?]
WARN 2018-11-07T21:56:04,866 (Worker thread '33') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,866 (Worker thread '22') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,872 (Worker thread '92') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,883 (Worker thread '37') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,883 (Worker thread '82') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,952 (Worker thread '13') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,953 (Worker thread '0') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,976 (Worker thread '47') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,976 (Worker thread '97') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:04,976 (Worker thread '84') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,004 (Worker thread '19') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,053 (Worker thread '16') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,079 (Worker thread '20') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,080 (Worker thread '76') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,114 (Worker thread '63') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,117 (Worker thread '93') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,143 (Worker thread '43') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,191 (Worker thread '9') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,191 (Worker thread '44') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,262 (Worker thread '99') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,276 (Worker thread '11') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,327 (Worker thread '48') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,353 (Worker thread '5') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,389 (Worker thread '31') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,414 (Worker thread '96') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,511 (Worker thread '65') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,514 (Worker thread '52') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,545 (Worker thread '39') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,561 (Worker thread '88') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,569 (Worker thread '72') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,640 (Worker thread '27') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,661 (Worker thread '51') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,765 (Worker thread '81') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,769 (Worker thread '2') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,787 (Worker thread '1') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,859 (Worker thread '83') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,886 (Worker thread '55') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,892 (Worker thread '3') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,910 (Worker thread '77') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,910 (Worker thread '40') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARN 2018-11-07T21:56:05,910 (Worker thread '32') - Service interruption reported for job 1533797717712 connection 'WinShare': Tika down, retrying: Connect to localhost:9998 [localhost/127.0.0.1<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=0gy%2FsM9DftXg%2BCKb66HCqY9lawxCr7K%2BPyaAA1vHSBk%3D&reserved=0>, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)




My Tika server is standalone and started with the option “-spawnChild”
Java -jar /opt/tika/tika-server-1.19.jar -spawnChild
This option, permits to Tika to restart a process f it crash for some cause as reported here:
As of Tika 1.19, users can make tika-server more robust by running it with the -spawnChild option. This starts tika-server in a child process, and if there's an OOM, a timeout or other catastrophic problem with the child process, the parent process will kill and/or restart the child process.

(https://wiki.apache.org/tika/TikaJAXRS<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.apache.org%2Ftika%2FTikaJAXRS&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ca3ad481a5390410a16a508d65481239d%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=VGujdJkXebuTfELhZLiOz31%2B9tAxS3c9QCBHQR5Rx2k%3D&reserved=0>)

I suppose, that manifold, when many times Tika server restart a child, it collects an interruption, and after many interruption the job stop.

After that, I need to start again the job.

Is there any way to workaround this?

Thanks a lot

Mario
Mime
View raw message