kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Yeargers <jon.yearg...@cedexis.com>
Subject Re: rocksdb error(s)
Date Mon, 12 Dec 2016 19:35:17 GMT
The attached error log shows several of the problems I've run into. After
hitting this list (and typically much less) the app dies.

On Mon, Dec 12, 2016 at 4:48 AM, Damian Guy <damian.guy@gmail.com> wrote:

> Just set the log level to debug and then run your app until you start
> seeing the problem.
> Thanks
>
> On Mon, 12 Dec 2016 at 12:47 Jon Yeargers <jon.yeargers@cedexis.com>
> wrote:
>
> > I can log whatever you need. Tell me what is useful.
> >
> > On Mon, Dec 12, 2016 at 4:43 AM, Damian Guy <damian.guy@gmail.com>
> wrote:
> >
> > > If you provide the logs from your streams application then we might
> have
> > > some chance of working out what is going on. Without logs then we
> really
> > > don't have much hope of diagnosing the problem.
> > >
> > > On Mon, 12 Dec 2016 at 12:18 Jon Yeargers <jon.yeargers@cedexis.com>
> > > wrote:
> > >
> > > > Im running as many threads as I have partitions on this topic. Just
> > > curious
> > > > if it would make any difference to the seemingly endless rebalancing
> > > woes.
> > > >
> > > > So far no change. In fact, I'll often see all 10 partitions (plus the
> > 2 x
> > > > 10 for the two aggregations) assigned to a single thread.
> > > >
> > > > On Mon, Dec 12, 2016 at 4:15 AM, Jon Yeargers <
> > jon.yeargers@cedexis.com>
> > > > wrote:
> > > >
> > > > > At this moment I have 5 instances each running 2 threads.
> > > > > Single instance / machine.
> > > > >
> > > > > Define 'full logs' ?
> > > > >
> > > > > On Mon, Dec 12, 2016 at 3:54 AM, Damian Guy <damian.guy@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Jon,
> > > > >>
> > > > >> How many StreamThreads do you have running?
> > > > >> How many application instances?
> > > > >> Do you have more than one instance per machine? If yes, are they
> > > sharing
> > > > >> the same State Directory?
> > > > >> Do you have full logs that can be provided so we can try and
see
> > > > how/what
> > > > >> is happening?
> > > > >>
> > > > >> Thanks,
> > > > >> Damian
> > > > >>
> > > > >> On Mon, 12 Dec 2016 at 10:17 Jon Yeargers <
> jon.yeargers@cedexis.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > No luck here. Moved all state storage to a non-tmp folder
and
> > > > restarted.
> > > > >> > Still hitting the 'No locks available' error quite frequently.
> > > > >> >
> > > > >> > On Sun, Dec 11, 2016 at 3:45 PM, Jon Yeargers <
> > > > jon.yeargers@cedexis.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I moved the state folder to a separate drive and linked
out to
> > it.
> > > > >> > >
> > > > >> > > I'll try your suggestion and point directly.
> > > > >> > >
> > > > >> > > On Sun, Dec 11, 2016 at 2:20 PM, Matthias J. Sax <
> > > > >> matthias@confluent.io>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> I am not sure, but this might be related with your
state
> > > directory.
> > > > >> > >>
> > > > >> > >> You use default directory that is located in /tmp
-- could it
> > be,
> > > > >> that
> > > > >> > >> /tmp gets clean up and thus you loose files/directories?
> > > > >> > >>
> > > > >> > >> Try to reconfigure your state directory via StreamsConfig:
> > > > >> > >> http://docs.confluent.io/current/streams/developer-guide.
> > > > >> > >> html#optional-configuration-parameters
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> -Matthias
> > > > >> > >>
> > > > >> > >> On 12/11/16 1:28 AM, Jon Yeargers wrote:
> > > > >> > >> > Seeing this appearing somewhat frequently
-
> > > > >> > >> >
> > > > >> > >> > org.apache.kafka.streams.errors.ProcessorStateException:
> > Error
> > > > >> opening
> > > > >> > >> > store minute_agg_stream-201612100812 at location
> > > > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_
> ag
> > > > >> > >> g_stream-201612100812
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.
> openDB
> > > > >> > >> (RocksDBStore.java:196)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.
> openDB
> > > > >> > >> (RocksDBStore.java:158)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore$
> > > > >> > >> Segment.openDB(RocksDBWindowStore.java:72)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore.
> > > > >> > >> getOrCreateSegment(RocksDBWindowStore.java:402)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore.
> > > > >> > >> putInternal(RocksDBWindowStore.java:333)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore.
> > > > >> > >> access$100(RocksDBWindowStore.java:51)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore$
> > > > >> > >> 2.restore(RocksDBWindowStore.java:212)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.
> ProcessorStateM
> > > > >> > >> anager.restoreActiveState(ProcessorStateManager.java:235)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.
> ProcessorStateM
> > > > >> > >> anager.register(ProcessorStateManager.java:198)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.
> ProcessorContex
> > > > >> > >> tImpl.register(ProcessorContextImpl.java:123)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> RocksDBWindowStore.
> > > > >> > >> init(RocksDBWindowStore.java:206)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> MeteredWindowStore.
> > > > >> > >> init(MeteredWindowStore.java:66)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.
> CachingWindowStore.
> > > > >> > >> init(CachingWindowStore.java:64)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.
> AbstractTask.in
> > > > >> > >> itializeStateStores(AbstractTask.java:81)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamTask.<
> > > > >> > >> init>(StreamTask.java:120)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > > >> > >> createStreamTask(StreamThread.java:633)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > > >> > >> addStreamTasks(StreamThread.java:660)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > > >> > >> access$100(StreamThread.java:69)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.
> StreamThread$1.
> > > > >> > >> onPartitionsAssigned(StreamThread.java:124)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.internals.
> ConsumerCoordina
> > > > >> > >> tor.onJoinComplete(ConsumerCoordinator.java:228)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.internals.
> AbstractCoordina
> > > > >> > >> tor.joinGroupIfNeeded(AbstractCoordinator.java:313)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.internals.
> AbstractCoordina
> > > > >> > >> tor.ensureActiveGroup(AbstractCoordinator.java:277)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.internals.
> ConsumerCoordina
> > > > >> > >> tor.poll(ConsumerCoordinator.java:259)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(
> > > > >> > >> KafkaConsumer.java:1013)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaCo
> > > > >> > >> nsumer.java:979)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > > >> > >> runLoop(StreamThread.java:407)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread.
> > > > >> > >> run(StreamThread.java:242)
> > > > >> > >> >
> > > > >> > >> > Caused by: org.rocksdb.RocksDBException: IO
error: lock
> > > > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_
> ag
> > > > >> > >> g_stream-201612100812/LOCK:
> > > > >> > >> > No locks available
> > > > >> > >> >
> > > > >> > >> >         at org.rocksdb.RocksDB.open(Native
Method)
> > > > >> > >> >
> > > > >> > >> >         at org.rocksdb.RocksDB.open(RocksDB.java:184)
> > > > >> > >> >
> > > > >> > >> >         at
> > > > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.
> openDB
> > > > >> > >> (RocksDBStore.java:189)
> > > > >> > >> >
> > > > >> > >> >         ... 26 common frames omitted
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message