nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Flow File Stuck for no reason
Date Tue, 06 Dec 2016 15:14:41 GMT
Hello

This part of the stack dump is the only thing that appears out of
place to me.  it appears to be attempting to read from the socket or
stuck there.  Perhaps you can use a more recent client library for
that custom HD processor?

"Timer-Driven Process Thread-8" Id=78 RUNNABLE  (in native code)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- waiting on java.io.BufferedInputStream@7b23ac8
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
- waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
at custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78

Thanks
Joe

On Tue, Dec 6, 2016 at 9:05 AM, Manish G <manish.ggoru@gmail.com> wrote:
> Hi,
>
> The issue happened again today. I restarted the NiFi and it started working
> again. Below are the rows from bootstrap.log after nifi.sh dump.
> Could this be because of some other process like diagnostic monitor on Azure
> storage blob?
>
> Thanks,
> Manish
>
>
> 2016-12-06 08:41:47,079 INFO [main] org.apache.nifi.bootstrap.RunNiFi "
> "Attach Listener" Id=6 RUNNABLE
>
> "AzureNativeFilesystemStore-UploadBandwidthUpdater" Id=94 TIMED_WAITING  on
> null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.azure.metrics.BandwidthGaugeUpdater$UploadBandwidthUpdater.run(BandwidthGaugeUpdater.java:266)
> at java.lang.Thread.run(Unknown Source)
>
> "Cleanup Archive for default" Id=54 RUNNABLE
> at sun.nio.fs.WindowsNativeDispatcher.FindClose(Native Method)
> at sun.nio.fs.WindowsDirectoryStream.close(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.checkAccess(Unknown Source)
> at java.nio.file.Files.exists(Unknown Source)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1307)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1546)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3cd89c72
>
> "DestroyJavaVM" Id=99 RUNNABLE
>
> "Event-Driven Process Thread-1" Id=56 TIMED_WAITING  on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@641ab175
>
> "Event-Driven Process Thread-2" Id=57 TIMED_WAITING  on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3bc33ad9
>
> "Event-Driven Process Thread-3" Id=58 TIMED_WAITING  on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@6b35b0f4
>
> "Event-Driven Process Thread-4" Id=59 TIMED_WAITING  on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@2cf669f2
>
> "Event-Driven Process Thread-5" Id=60 TIMED_WAITING  on
> java.lang.Object@37b2da15
> at java.lang.Object.wait(Native Method)
> at
> org.apache.nifi.controller.EventDrivenWorkerQueue.poll(EventDrivenWorkerQueue.java:70)
> at
> org.apache.nifi.controller.scheduling.EventDrivenSchedulingAgent$EventDrivenTask.run(EventDrivenSchedulingAgent.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@1dd6e400
>
> "FileSystemRepository Workers Thread-1" Id=52 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-2" Id=53 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-3" Id=65 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "FileSystemRepository Workers Thread-4" Id=66 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7acfd339
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Finalizer" Id=3 WAITING  on java.lang.ref.ReferenceQueue$Lock@12de219
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(Unknown Source)
> at java.lang.ref.ReferenceQueue.remove(Unknown Source)
> at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
>
> "Flow Service Tasks Thread-1" Id=63 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Flow Service Tasks Thread-2" Id=64 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cbdc2d2
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-1" Id=61 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-2" Id=100 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-3" Id=101 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Framework Task Thread Thread-4" Id=108 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2caaa557
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 File Lock Watchdog D:/nififiles/database_repository/nifi-audit.lock.db"
> Id=44 TIMED_WAITING  on null
> at java.lang.Thread.sleep(Native Method)
> at org.h2.store.FileLock.run(FileLock.java:517)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 File Lock Watchdog D:/nififiles/database_repository/nifi-users.lock.db"
> Id=42 TIMED_WAITING  on null
> at java.lang.Thread.sleep(Native Method)
> at org.h2.store.FileLock.run(FileLock.java:517)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 Log Writer NIFI-AUDIT" Id=45 TIMED_WAITING  on
> org.h2.store.WriterThread@3bbc1f86
> at java.lang.Object.wait(Native Method)
> at org.h2.store.WriterThread.run(WriterThread.java:103)
> at java.lang.Thread.run(Unknown Source)
>
> "H2 Log Writer NIFI-USERS" Id=43 TIMED_WAITING  on
> org.h2.store.WriterThread@1a7f0896
> at java.lang.Object.wait(Native Method)
> at org.h2.store.WriterThread.run(WriterThread.java:103)
> at java.lang.Thread.run(Unknown Source)
>
> "Listen to Bootstrap" Id=22 RUNNABLE
> at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
> at java.net.PlainSocketImpl.accept(Unknown Source)
> - waiting on java.net.SocksSocketImpl@2d73cb7e
> at java.net.ServerSocket.implAccept(Unknown Source)
> at java.net.ServerSocket.accept(Unknown Source)
> at
> org.apache.nifi.BootstrapListener$Listener.run(BootstrapListener.java:158)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-27-selector-ServerConnectorManager@2086a3b2/0" Id=27
> RUNNABLE  (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@1b46e697
> - waiting on java.util.Collections$UnmodifiableSet@23c7c131
> - waiting on sun.nio.ch.WindowsSelectorImpl@8bde1da
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-28-selector-ServerConnectorManager@2086a3b2/1" Id=28
> RUNNABLE  (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@20930553
> - waiting on java.util.Collections$UnmodifiableSet@7dfac566
> - waiting on sun.nio.ch.WindowsSelectorImpl@472c48e3
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-29-selector-ServerConnectorManager@2086a3b2/2" Id=29
> RUNNABLE  (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@6f765847
> - waiting on java.util.Collections$UnmodifiableSet@681a4ed4
> - waiting on sun.nio.ch.WindowsSelectorImpl@48f28891
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-30-selector-ServerConnectorManager@2086a3b2/3" Id=30
> RUNNABLE  (in native code)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> - waiting on sun.nio.ch.Util$2@5fab180d
> - waiting on java.util.Collections$UnmodifiableSet@31922b45
> - waiting on sun.nio.ch.WindowsSelectorImpl@54a785d1
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
> at
> org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web
> Server-31-acceptor-0@236f25b5-ServerConnector@403230be{HTTP/1.1}{0.0.0.0:8080}"
> Id=31 RUNNABLE  (in native code)
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> - waiting on java.lang.Object@5c7d5af
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> - waiting on java.lang.Object@5c7d5af
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web
> Server-32-acceptor-1@1657bbfa-ServerConnector@403230be{HTTP/1.1}{0.0.0.0:8080}"
> Id=32 BLOCKED  on java.lang.Object@5c7d5af
> at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
> at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43974" Id=43974 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43979" Id=43979 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "NiFi Web Server-43987" Id=43987 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61cb9802
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@36920bd6Timer" Id=35
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7192984e
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@3a7e365Timer" Id=40
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a655aac
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@3c9c6245Timer" Id=38
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57f47aa1
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@45adc393Timer" Id=36
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@52d7f52
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@5853431fTimer" Id=97
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40d96d76
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@6c90063fTimer" Id=98
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542a5a90
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@71adfeddTimer" Id=39
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c6f1485
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@7c663eafTimer" Id=37
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35ade371
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "org.eclipse.jetty.server.session.HashSessionManager@b6182cbTimer" Id=95
> TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39a0e827
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "pool-1-thread-1" Id=43988 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f0c7b93
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "pool-1-thread-2" Id=43989 RUNNABLE
> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
> at org.apache.nifi.BootstrapListener.writeDump(BootstrapListener.java:221)
> at org.apache.nifi.BootstrapListener.access$500(BootstrapListener.java:49)
> at
> org.apache.nifi.BootstrapListener$Listener$1.run(BootstrapListener.java:196)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@162f0c7
>
> "pool-16-thread-1" Id=69 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47f9a0c5
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-1" Id=48 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-2" Id=49 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Maintenance Thread-3" Id=50 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3995f134
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Query Thread-1" Id=35981 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@65f410e3
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Repository Rollover Thread-1" Id=116 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Provenance Repository Rollover Thread-2" Id=132 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20f69239
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Reference Handler" Id=2 WAITING  on java.lang.ref.Reference$Lock@1cfe6544
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Unknown Source)
> at java.lang.ref.Reference.tryHandlePending(Unknown Source)
> at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
>
> "Scheduler-1822651119" Id=102 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@119c3348
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Signal Dispatcher" Id=5 RUNNABLE
>
> "StandardProcessScheduler Thread-1" Id=73 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-1" Id=67 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-2" Id=82 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-2" Id=68 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-3" Id=84 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-3" Id=72 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-4" Id=86 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-4" Id=81 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-5" Id=88 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-5" Id=83 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-6" Id=90 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-6" Id=85 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-7" Id=91 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-7" Id=87 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-8" Id=92 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2eb758
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "StandardProcessScheduler Thread-8" Id=89 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4497d116
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer for 'azure-file-system' metrics system" Id=93 TIMED_WAITING  on
> java.util.TaskQueue@49d69a28
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Unknown Source)
> at java.util.TimerThread.run(Unknown Source)
>
> "Timer-Driven Process Thread-1" Id=62 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-10" Id=80 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-2" Id=70 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-3" Id=71 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-4" Id=74 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-5" Id=75 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-6" Id=76 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-7" Id=77 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Timer-Driven Process Thread-8" Id=78 RUNNABLE  (in native code)
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.io.BufferedInputStream.fill(Unknown Source)
> at java.io.BufferedInputStream.read1(Unknown Source)
> at java.io.BufferedInputStream.read(Unknown Source)
> - waiting on java.io.BufferedInputStream@7b23ac8
> at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
> at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown
> Source)
> - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown
> Source)
> - waiting on sun.net.www.protocol.http.HttpURLConnection@3d1cc911
> at java.net.HttpURLConnection.getResponseCode(Unknown Source)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:124)
> at
> com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:1638)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1603)
> at
> custom.nifi.processor.hdfs.PutHDFSHDCustom.onTrigger(PutHDFSHDCustom.java:247)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@3b501d78
>
> "Timer-Driven Process Thread-9" Id=79 WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a48e873
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> "Write-Ahead Local State Provider Maintenance" Id=55 TIMED_WAITING  on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc7abae
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> "
>
>
> On Thu, Dec 1, 2016 at 12:02 PM, Manish G <manish.ggoru@gmail.com> wrote:
>>
>> Got it. I was not aware of "bin/nifi.sh dump". I will try that once I see
>> the issue again.
>>
>> Thanks,
>> Manish
>>
>> On Thu, Dec 1, 2016 at 11:59 AM, Joe Witt <joe.witt@gmail.com> wrote:
>>>
>>> Manish
>>>
>>> When it is in the stuck state can you please run
>>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log that
>>> would aid us in narrowing in on a possible cause.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 1, 2016 at 8:44 AM, Manish G <manish.ggoru@gmail.com> wrote:
>>> > Hi Joe,
>>> >
>>> > Here is what I can see in the App Log:
>>> >
>>> > 2016-12-01 09:28:52,004 ERROR [Timer-Driven Process Thread-4]
>>> > testing.nifi.processor.hdfs.testingPutHDFS ""
>>> > org.apache.hadoop.fs.azure.AzureException:
>>> > java.util.NoSuchElementException:
>>> > An error occurred while enumerating the result, check the original
>>> > exception
>>> > for details.
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1930)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1592)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
>>> > ~[hadoop-common-2.7.2.jar:na]
>>> > at
>>> >
>>> > testing.nifi.processor.hdfs.testingPutHDFS.onTrigger(testingPutHDFS.java:260)
>>> > ~[testing.nifi.processor-1.0.0.nar-unpacked/:na]
>>> > at
>>> >
>>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> > [nifi-api-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at
>>> >
>>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>> > [nifi-framework-core-0.7.0.jar:0.7.0]
>>> > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at
>>> >
>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>>> > Source) [na:1.8.0_101]
>>> > at
>>> >
>>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> > Source) [na:1.8.0_101]
>>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>> > [na:1.8.0_101]
>>> > at java.lang.Thread.run(Unknown Source) [na:1.8.0_101]
>>> > Caused by: java.util.NoSuchElementException: An error occurred while
>>> > enumerating the result, check the original exception for details.
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:128)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > at
>>> >
>>> > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1909)
>>> > ~[hadoop-azure-2.7.2.jar:na]
>>> > ... 15 common frames omitted
>>> > Caused by: com.microsoft.azure.storage.StorageException: Forbidden
>>> > at
>>> >
>>> > com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:202)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:172)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:273)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > ... 17 common frames omitted
>>> > Caused by: java.lang.NullPointerException: null
>>> > at
>>> >
>>> > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:181)
>>> > ~[azure-storage-2.0.0.jar:na]
>>> > ... 18 common frames omitted
>>> >
>>> > Regards,
>>> > Manish
>>> >
>>> > On Thu, Dec 1, 2016 at 10:36 AM, Joe Witt <joe.witt@gmail.com> wrote:
>>> >>
>>> >> Manish
>>> >>
>>> >> Please produce and share the thread dump I mentioned.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Dec 1, 2016 7:23 AM, "Manish G" <manish.ggoru@gmail.com> wrote:
>>> >>>
>>> >>> Hi,
>>> >>>
>>> >>> I don't know why, but this is happening now more frequently. Where
>>> >>> should
>>> >>> I look into to find the root cause?
>>> >>>
>>> >>> Thanks,
>>> >>> Manish
>>> >>>
>>> >>> On Wed, Nov 30, 2016 at 9:20 PM, Manish G <manish.ggoru@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi Joe,
>>> >>>>
>>> >>>> Thanks for the quick reply. Yes, the processor keeps running on a
>>> >>>> single
>>> >>>> thread (even after stopping). And the number remains there even
>>> >>>> after
>>> >>>> stopping.
>>> >>>> Today, it happened on my customized putHDFS processor. Only thing
>>> >>>> different in this processor is - I have added an additional
>>> >>>> attribute that
>>> >>>> tells if the processor created the directory while loading the file
>>> >>>> on HDFS.
>>> >>>> I don't think this should be the issue though.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Manish
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Nov 30, 2016 at 7:05 PM, Joe Witt <joe.witt@gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Manish
>>> >>>>>
>>> >>>>> When it is stuck do you see a number in the top right corner of the
>>> >>>>> processor?  When you stop it does the number remain?  That number
>>> >>>>> is
>>> >>>>> telling you how many threads are still executing.  Which processor
>>> >>>>> are
>>> >>>>> we talking about?  When it is in the stuck state can you please run
>>> >>>>> bin/nifi.sh dump.  If you can then share the nifi-bootstrap.log
>>> >>>>> that
>>> >>>>> would aid us in narrowing in on a possible cause.
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>> Joe
>>> >>>>>
>>> >>>>> On Wed, Nov 30, 2016 at 7:02 PM, Manish G <manish.ggoru@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >
>>> >>>>> > Hi,
>>> >>>>> >
>>> >>>>> > I have noticed that sometime a flow file gets stuck on a
>>> >>>>> > processor
>>> >>>>> > for a
>>> >>>>> > very long time for no reason and then I can not even stop the
>>> >>>>> > processor to
>>> >>>>> > look at the flow flow file from queue. If I click on stop, then
>>> >>>>> > processor
>>> >>>>> > goes into a state where I cannot start/stop the processor.
>>> >>>>> >
>>> >>>>> > On restarting the NiFi, the file gets processed successfully and
>>> >>>>> > routed to
>>> >>>>> > success queue. I checked in App log, but everything seems to be
>>> >>>>> > normal for
>>> >>>>> > the flow file. I don't see anything mysterious in provenance too
>>> >>>>> > (except
>>> >>>>> > that queue time is in hours).
>>> >>>>> >
>>> >>>>> > Has anyone else faced a similar issue? What else should I check
>>> >>>>> > to
>>> >>>>> > identify
>>> >>>>> > the root cause for this?
>>> >>>>> >
>>> >>>>> > Thanks,
>>> >>>>> > Manish
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>>
>>> >>>>
>>> >>>> With Warm Regards,
>>> >>>> Manish
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>>
>>> >>>
>>> >>> With Warm Regards,
>>> >>> Manish
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> >
>>> > With Warm Regards,
>>> > Manish
>>
>>
>>
>>
>> --
>>
>>
>> With Warm Regards,
>> Manish
>
>
>
>
> --
>
>
> With Warm Regards,
> Manish

Mime
View raw message