kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: rocksdb error(s)
Date Mon, 12 Dec 2016 12:48:58 GMT
Just set the log level to debug and then run your app until you start
seeing the problem.
Thanks

On Mon, 12 Dec 2016 at 12:47 Jon Yeargers <jon.yeargers@cedexis.com> wrote:

> I can log whatever you need. Tell me what is useful.
>
> On Mon, Dec 12, 2016 at 4:43 AM, Damian Guy <damian.guy@gmail.com> wrote:
>
> > If you provide the logs from your streams application then we might have
> > some chance of working out what is going on. Without logs then we really
> > don't have much hope of diagnosing the problem.
> >
> > On Mon, 12 Dec 2016 at 12:18 Jon Yeargers <jon.yeargers@cedexis.com>
> > wrote:
> >
> > > Im running as many threads as I have partitions on this topic. Just
> > curious
> > > if it would make any difference to the seemingly endless rebalancing
> > woes.
> > >
> > > So far no change. In fact, I'll often see all 10 partitions (plus the
> 2 x
> > > 10 for the two aggregations) assigned to a single thread.
> > >
> > > On Mon, Dec 12, 2016 at 4:15 AM, Jon Yeargers <
> jon.yeargers@cedexis.com>
> > > wrote:
> > >
> > > > At this moment I have 5 instances each running 2 threads.
> > > > Single instance / machine.
> > > >
> > > > Define 'full logs' ?
> > > >
> > > > On Mon, Dec 12, 2016 at 3:54 AM, Damian Guy <damian.guy@gmail.com>
> > > wrote:
> > > >
> > > >> Jon,
> > > >>
> > > >> How many StreamThreads do you have running?
> > > >> How many application instances?
> > > >> Do you have more than one instance per machine? If yes, are they
> > sharing
> > > >> the same State Directory?
> > > >> Do you have full logs that can be provided so we can try and see
> > > how/what
> > > >> is happening?
> > > >>
> > > >> Thanks,
> > > >> Damian
> > > >>
> > > >> On Mon, 12 Dec 2016 at 10:17 Jon Yeargers <jon.yeargers@cedexis.com
> >
> > > >> wrote:
> > > >>
> > > >> > No luck here. Moved all state storage to a non-tmp folder and
> > > restarted.
> > > >> > Still hitting the 'No locks available' error quite frequently.
> > > >> >
> > > >> > On Sun, Dec 11, 2016 at 3:45 PM, Jon Yeargers <
> > > jon.yeargers@cedexis.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > I moved the state folder to a separate drive and linked
out to
> it.
> > > >> > >
> > > >> > > I'll try your suggestion and point directly.
> > > >> > >
> > > >> > > On Sun, Dec 11, 2016 at 2:20 PM, Matthias J. Sax <
> > > >> matthias@confluent.io>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> I am not sure, but this might be related with your state
> > directory.
> > > >> > >>
> > > >> > >> You use default directory that is located in /tmp --
could it
> be,
> > > >> that
> > > >> > >> /tmp gets clean up and thus you loose files/directories?
> > > >> > >>
> > > >> > >> Try to reconfigure your state directory via StreamsConfig:
> > > >> > >> http://docs.confluent.io/current/streams/developer-guide.
> > > >> > >> html#optional-configuration-parameters
> > > >> > >>
> > > >> > >>
> > > >> > >> -Matthias
> > > >> > >>
> > > >> > >> On 12/11/16 1:28 AM, Jon Yeargers wrote:
> > > >> > >> > Seeing this appearing somewhat frequently -
> > > >> > >> >
> > > >> > >> > org.apache.kafka.streams.errors.ProcessorStateException:
> Error
> > > >> opening
> > > >> > >> > store minute_agg_stream-201612100812 at location
> > > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag
> > > >> > >> g_stream-201612100812
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB
> > > >> > >> (RocksDBStore.java:196)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB
> > > >> > >> (RocksDBStore.java:158)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$
> > > >> > >> Segment.openDB(RocksDBWindowStore.java:72)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> > > >> > >> getOrCreateSegment(RocksDBWindowStore.java:402)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> > > >> > >> putInternal(RocksDBWindowStore.java:333)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> > > >> > >> access$100(RocksDBWindowStore.java:51)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$
> > > >> > >> 2.restore(RocksDBWindowStore.java:212)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM
> > > >> > >> anager.restoreActiveState(ProcessorStateManager.java:235)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM
> > > >> > >> anager.register(ProcessorStateManager.java:198)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorContex
> > > >> > >> tImpl.register(ProcessorContextImpl.java:123)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> > > >> > >> init(RocksDBWindowStore.java:206)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.MeteredWindowStore.
> > > >> > >> init(MeteredWindowStore.java:66)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.CachingWindowStore.
> > > >> > >> init(CachingWindowStore.java:64)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.AbstractTask.in
> > > >> > >> itializeStateStores(AbstractTask.java:81)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamTask.<
> > > >> > >> init>(StreamTask.java:120)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > >> > >> createStreamTask(StreamThread.java:633)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > >> > >> addStreamTasks(StreamThread.java:660)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > >> > >> access$100(StreamThread.java:69)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread$1.
> > > >> > >> onPartitionsAssigned(StreamThread.java:124)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina
> > > >> > >> tor.onJoinComplete(ConsumerCoordinator.java:228)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina
> > > >> > >> tor.joinGroupIfNeeded(AbstractCoordinator.java:313)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina
> > > >> > >> tor.ensureActiveGroup(AbstractCoordinator.java:277)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina
> > > >> > >> tor.poll(ConsumerCoordinator.java:259)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(
> > > >> > >> KafkaConsumer.java:1013)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaCo
> > > >> > >> nsumer.java:979)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > >> > >> runLoop(StreamThread.java:407)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > >> > >> run(StreamThread.java:242)
> > > >> > >> >
> > > >> > >> > Caused by: org.rocksdb.RocksDBException: IO error:
lock
> > > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag
> > > >> > >> g_stream-201612100812/LOCK:
> > > >> > >> > No locks available
> > > >> > >> >
> > > >> > >> >         at org.rocksdb.RocksDB.open(Native Method)
> > > >> > >> >
> > > >> > >> >         at org.rocksdb.RocksDB.open(RocksDB.java:184)
> > > >> > >> >
> > > >> > >> >         at
> > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB
> > > >> > >> (RocksDBStore.java:189)
> > > >> > >> >
> > > >> > >> >         ... 26 common frames omitted
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message