kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhavesh Patel <bhavesh3...@gmail.com>
Subject Re: Kafka stream issue : Deleting obsolete state directory
Date Mon, 24 Sep 2018 15:10:42 GMT
Hi Bill,

Please find the log as requested.

2018-09-24 15:53:59.594  INFO 1 --- [pool-2-thread-1]
i.micrometer.influx.InfluxMeterRegistry  : successfully sent 2793 metrics
to influx

2018-09-24 15:54:59.608  INFO 1 --- [pool-2-thread-1]
i.micrometer.influx.InfluxMeterRegistry  : successfully sent 2793 metrics
to influx

2018-09-24 15:55:15.957  INFO 1 --- [f-CleanupThread]
o.a.k.s.p.internals.StateDirectory       : stream-thread
[ApplicationName-ca4fb977-8029-4e4e-b733-2c1e4bd31d6f-CleanupThread]
Deleting obsolete state directory 0_0 for task 0_0 as 1096957ms has elapsed
(cleanup delay is 600000ms).

2018-09-24 15:55:15.975 ERROR 1 --- [f-CleanupThread]
o.a.k.s.p.internals.StateDirectory       : stream-thread
[ApplicationName-ca4fb977-8029-4e4e-b733-2c1e4bd31d6f-CleanupThread] Failed
to delete the state directory.



java.nio.file.DirectoryNotEmptyException:
/tmp/ApplicationName/ApplicationName/0_0

                at
sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
~[na:1.8.0_161]

                at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.8.0_161]

                at java.nio.file.Files.delete(Files.java:1126)
~[na:1.8.0_161]

                at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:740)
~[kafka-clients-2.0.0.jar!/:na]

                at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:723)
~[kafka-clients-2.0.0.jar!/:na]

                at java.nio.file.Files.walkFileTree(Files.java:2688)
~[na:1.8.0_161]

                at java.nio.file.Files.walkFileTree(Files.java:2742)
~[na:1.8.0_161]

                at
org.apache.kafka.common.utils.Utils.delete(Utils.java:723)
~[kafka-clients-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287)
[kafka-streams-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:250)
[kafka-streams-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.KafkaStreams$2.run(KafkaStreams.java:800)
[kafka-streams-2.0.0.jar!/:na]

                at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_161]

                at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_161]

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_161]

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_161]

                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]

                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]

                at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]



2018-09-24 15:55:15.981  INFO 1 --- [f-CleanupThread]
o.a.k.s.p.internals.StateDirectory       : stream-thread
[ApplicationName-ca4fb977-8029-4e4e-b733-2c1e4bd31d6f-CleanupThread]
Deleting obsolete state directory 0_1 for task 0_1 as 1096981ms has elapsed
(cleanup delay is 600000ms).

2018-09-24 15:55:15.985 ERROR 1 --- [f-CleanupThread]
o.a.k.s.p.internals.StateDirectory       : stream-thread
[ApplicationName-ca4fb977-8029-4e4e-b733-2c1e4bd31d6f-CleanupThread] Failed
to delete the state directory.



java.nio.file.DirectoryNotEmptyException:
/tmp/ApplicationName/ApplicationName/0_1

                at
sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
~[na:1.8.0_161]

                at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.8.0_161]

                at java.nio.file.Files.delete(Files.java:1126)
~[na:1.8.0_161]

                at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:740)
~[kafka-clients-2.0.0.jar!/:na]

                at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:723)
~[kafka-clients-2.0.0.jar!/:na]

                at java.nio.file.Files.walkFileTree(Files.java:2688)
~[na:1.8.0_161]

                at java.nio.file.Files.walkFileTree(Files.java:2742)
~[na:1.8.0_161]

                at
org.apache.kafka.common.utils.Utils.delete(Utils.java:723)
~[kafka-clients-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287)
[kafka-streams-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:250)
[kafka-streams-2.0.0.jar!/:na]

                at
org.apache.kafka.streams.KafkaStreams$2.run(KafkaStreams.java:800)
[kafka-streams-2.0.0.jar!/:na]

                at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_161]

                at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_161]

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_161]

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_161]

                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]

                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]

                at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]



On Mon, Sep 24, 2018, 14:19 Bill Bejeck <bill@confluent.io> wrote:

> Hi Bhavesh,
>
> I've taken a look at what you've posted, but I'll need the logs *before
> *the
> exception occurred.  Can you attach the entire log?
>
> In the meantime, if this causing issues, you can effectively disable
> *state.cleanup.delay.ms
> <http://state.cleanup.delay.ms> *to a large value (Long.MAX_VALUE ?)
>
> Thanks,
> Bill
>
> On Thu, Sep 20, 2018 at 4:56 PM Bhavesh Patel <bhavesh3184@gmail.com>
> wrote:
>
> > Hi Bill,
> >
> > Yes and thanks for your response.
> >
> > Many thanks,
> > Bhavesh Patel
> >
> > On Thu, Sep 20, 2018, 21:31 Bill Bejeck <bill@confluent.io> wrote:
> >
> > > Hi Bhavesh,
> > >
> > > I haven't forgotten about you, just got sidetracked with some other
> > > issues.
> > >
> > > Just to be clear you are using Kafka Streams version 2.0 correct?
> > >
> > > I'll take a look in the next day or so.
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Thu, Sep 20, 2018 at 2:31 PM Bhavesh Patel <bhavesh3184@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > Is it possible to get h lp with below issue?
> > > >
> > > > Many Thanks,
> > > > Bhavesh Patel
> > > >
> > > > On Wed, Sep 19, 2018, 09:13 Bhavesh Patel <bhavesh3184@gmail.com>
> > wrote:
> > > >
> > > > > Hi Bill,
> > > > > Apologies, I was trying to attached image.
> > > > > Please find below the error log,
> > > > >
> > > > > 2018-09-18 09:26:09.112  INFO 1 --- [5-CleanupThread]
> > > > > o.a.k.s.p.internals.StateDirectory       : stream-thread
> > > > >
> [ApplicationName-1ae22d38-32d3-451a-b039-372c79b2e6a5-CleanupThread]
> > > > > Deleting obsolete state directory 2_1 for task 2_1 as 601112ms has
> > > > elapsed
> > > > > (cleanup delay is 600000ms).
> > > > >
> > > > > 2018-09-18 09:26:09.116 ERROR 1 --- [5-CleanupThread]
> > > > > o.a.k.s.p.internals.StateDirectory       : stream-thread
> > > > >
> [ApplicationName-1ae22d38-32d3-451a-b039-372c79b2e6a5-CleanupThread]
> > > > Failed
> > > > > to delete the state directory.
> > > > >
> > > > >
> > > > >
> > > > > java.nio.file.DirectoryNotEmptyException:
> > > > > /tmp/ApplicationName/ApplicationName/2_1
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> > > > > ~[na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> > > > > ~[na:1.8.0_161]
> > > > >
> > > > >                 at java.nio.file.Files.delete(Files.java:1126)
> > > > > ~[na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > >
> org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:740)
> > > > > ~[kafka-clients-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> > > > >
> > >
> org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:723)
> > > > > ~[kafka-clients-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> java.nio.file.Files.walkFileTree(Files.java:2688)
> > > > > ~[na:1.8.0_161]
> > > > >
> > > > >                 at
> java.nio.file.Files.walkFileTree(Files.java:2742)
> > > > > ~[na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > > org.apache.kafka.common.utils.Utils.delete(Utils.java:723)
> > > > > ~[kafka-clients-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287)
> > > > > [kafka-streams-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:250)
> > > > > [kafka-streams-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> > > > > org.apache.kafka.streams.KafkaStreams$2.run(KafkaStreams.java:800)
> > > > > [kafka-streams-2.0.0.jar!/:na]
> > > > >
> > > > >                 at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > [na:1.8.0_161]
> > > > >
> > > > >                 at java.lang.Thread.run(Thread.java:748)
> > [na:1.8.0_161]
> > > > >
> > > > > Many Thanks,
> > > > > Bhavesh Patel
> > > > >
> > > > >
> > > > > On Wed, Sep 19, 2018, 02:02 Bill Bejeck <bill@confluent.io>
wrote:
> > > > >
> > > > >> Hi Bhavesh,
> > > > >>
> > > > >> Sorry, but I'm still not seeing any log file attachment, maybe
> it's
> > > > being
> > > > >> filtered out.
> > > > >>
> > > > >> Can you copy the relevant section in the body of the email?
> > > > >>
> > > > >> Thanks,
> > > > >> Bill
> > > > >>
> > > > >> On Tue, Sep 18, 2018 at 3:21 PM Bhavesh Patel <
> > bhavesh3184@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Bill,
> > > > >> >
> > > > >> > Apologies, Please find the error stack attached.
> > > > >> >
> > > > >> > We are using kafka 2.0.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Bhavesh Patel
> > > > >> >
> > > > >> > On Tue, Sep 18, 2018 at 8:04 PM Bill Bejeck <bill@confluent.io>
> > > > wrote:
> > > > >> >
> > > > >> >> Hi Bhavesh,
> > > > >> >>
> > > > >> >> I don't see the log file attachment, can you resend
it?
> > > > >> >>
> > > > >> >> Also, what version of Kafka Streams are you running?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> Bill
> > > > >> >>
> > > > >> >> On Tue, Sep 18, 2018 at 12:31 PM Bhavesh Patel <
> > > > bhavesh3184@gmail.com>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Hi,
> > > > >> >> >
> > > > >> >> > We have a stream application where we are facing
a little
> > strange
> > > > >> >> problem
> > > > >> >> > with deleting state directory.
> > > > >> >> >
> > > > >> >> > Our application is deployed into in-house container
platform
> > and
> > > > >> >> storage is
> > > > >> >> > backed by nfs.
> > > > >> >> >
> > > > >> >> > When app is running for a while and when it hits
the cleanup
> > > delay
> > > > of
> > > > >> >> > 600000ms as defined dafault in Kafka settings we
are getting
> > > > >> exception
> > > > >> >> of
> > > > >> >> > DIRECTORYNOTEMPTYEXCEPTION.
> > > > >> >> >
> > > > >> >> > I am attaching the log below, thoughts on this
issue will be
> > much
> > > > >> >> > appreciated.
> > > > >> >> >
> > > > >> >> > Many Thanks,
> > > > >> >> > Bhavesh Patel
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message