kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sophie Blee-Goldman <sop...@confluent.io>
Subject Re: Kafka streams (2.1.1) - org.rocksdb.RocksDBException:Too many open files
Date Wed, 03 Jul 2019 18:20:00 GMT
Also, when you say you changed the OS limit are you referring to the system
limit or the user limit or both? If you increased one but not the other you
may still be hitting the lower limit.

On Wed, Jul 3, 2019 at 1:53 AM Patrik Kleindl <pkleindl@gmail.com> wrote:

> Hi
> Try to set it really low like Sophie suggested.
> You can verify if the settings take effect by checking the files in the
> rocksdb directories, I think it should be somewhere in OPTIONS or LOG
> br, Patrik
>
> > Am 03.07.2019 um 09:37 schrieb Thameem Ansari <thameema@gmail.com>:
> >
> > Tried setting the open files to 100 and 50 but the results are same. I
> checked the total open files while the streaming application was busy
> running just before getting the “too many open files” message it was around
> 41756 which is same as what we have got when we set to -1.
> >
> > VisualVM shows that there is no abnormality with the threads / memory or
> heap.
> >
> > Thanks
> > Thameem
> >
> >> On Jul 3, 2019, at 11:50 AM, Sophie Blee-Goldman <sophie@confluent.io>
> wrote:
> >>
> >> How sure are you that the open file count never goes beyond 50K? Are
> those
> >> numbers just from a snapshot after it crashed? It's possible rocks is
> >> creating a large number of files just for a short period of time (maybe
> >> while compacting) that causes the open file count to spike and go back
> down.
> >>
> >> For things to try, you should set the rocks config max.open.files to
> >> something less than infinity...if you're OS limit is 1 million and you
> have
> >> (rounding up) ~5k rocks instances, set this to 1 million / 5k = 200. If
> you
> >> set a lower limit and still hit this error, we can go from there
> >>
> >> On Tue, Jul 2, 2019 at 11:10 PM emailtokirank@gmail.com <
> >> emailtokirank@gmail.com> wrote:
> >>
> >>>
> >>>
> >>>> On 2019/07/03 05:46:45, Sophie Blee-Goldman <sophie@confluent.io>
> wrote:
> >>>> It sounds like rocksdb *is* honoring your configs -- the
> max.open.files
> >>>> config is an internal restriction that tells rocksdb how many open
> files
> >>> it
> >>>> is allowed to have, so if that's set to -1 (infinite) it won't ever
> try
> >>> to
> >>>> limit its open files and you may hit the OS limit.
> >>>>
> >>>> Think of it this way: if you have 100 rocksdb instances and a OS
> limit of
> >>>> 500, you should set max.open.files to 5  to avoid hitting this limit
> >>>> (assuming there are no other open files on the system, in reality
> you'd
> >>>> want some extra room there)
> >>>>
> >>>> On Tue, Jul 2, 2019 at 7:53 PM emailtokirank@gmail.com <
> >>>> emailtokirank@gmail.com> wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>>> On 2019/06/28 23:29:16, John Roesler <john@confluent.io>
wrote:
> >>>>>> Hey all,
> >>>>>>
> >>>>>> If you want to figure it out theoretically, if you print out
the
> >>>>>> topology description, you'll have some number of state stores
listed
> >>>>>> in there. The number of Rocks instances should just be
> >>>>>> (#global_state_stores +
> >>>>>> sum(#partitions_of_topic_per_local_state_store)) . The number
of
> >>>>>> stream threads isn't relevant here.
> >>>>>>
> >>>>>> You can also figure it out empirically: the first level of
> >>>>>> subdirectories in the state dir are Tasks, and then within that,
the
> >>>>>> next level is Stores. You should see the store directory names
match
> >>>>>> up with the stores listed in the topology description. The number
of
> >>>>>> Store directories is exactly the number of RocksDB instances
you
> >>> have.
> >>>>>>
> >>>>>> There are also metrics corresponding to each of the state stores,
so
> >>>>>> you can compute it from what you find in the metrics.
> >>>>>>
> >>>>>> Hope that helps,
> >>>>>> -john
> >>>>>>
> >>>>>> On Thu, Jun 27, 2019 at 6:46 AM Patrik Kleindl <pkleindl@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Kiran
> >>>>>>> Without much research my guess would be "num_stream_threads
*
> >>>>>>> (#global_state_stores +
> >>>>> sum(#partitions_of_topic_per_local_state_store))"
> >>>>>>> So 10 stores (regardless if explicitly defined or implicitely
> >>> because
> >>>>> of
> >>>>>>> some stateful operation) with 10 partitions each should
result in
> >>> 100
> >>>>>>> Rocksdb instances if you are running at the default of
> >>>>> num_stream_threads=1.
> >>>>>>>
> >>>>>>> As I wrote before, start with 100.
> >>>>>>> If the error persists, half the number, if not, double it
;-)
> >>> Repeat as
> >>>>>>> needed.
> >>>>>>>
> >>>>>>> If you reach the single-digit-range and the error still
shows up,
> >>> start
> >>>>>>> searching for any iterators over a store you might not have
closed.
> >>>>>>>
> >>>>>>> br, Patrik
> >>>>>>>
> >>>>>>> On Thu, 27 Jun 2019 at 13:11, emailtokirank@gmail.com <
> >>>>>>> emailtokirank@gmail.com> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2019/06/27 09:02:39, Patrik Kleindl <pkleindl@gmail.com>
> >>> wrote:
> >>>>>>>>> Hello Kiran
> >>>>>>>>>
> >>>>>>>>> First, the value for maxOpenFiles is per RocksDB
instance, and
> >>> the
> >>>>> number
> >>>>>>>>> of those can get high if you have a lot of topic
partitions
> >>> etc.
> >>>>>>>>> Check the directory (state dir) to see how many
there are.
> >>>>>>>>> Start with a low value (100) and see if that has
some effect.
> >>>>>>>>>
> >>>>>>>>> Second, because I just found out, you should use
> >>>>>>>>> BlockBasedTableConfig tableConfig = (BlockBasedTableConfig)
> >>>>>>>>> options.tableFormatConfig();
> >>>>>>>>>       tableConfig.setBlockCacheSize(100*1024*1024L);
> >>>>>>>>>       tableConfig.setBlockSize(8*1024L);
> >>>>>>>>> instead of creating a new object to prevent accidently
messing
> >>> up
> >>>>>>>>> references.
> >>>>>>>>>
> >>>>>>>>> Hope that helps
> >>>>>>>>> best regards
> >>>>>>>>> Patrik
> >>>>>>>>>
> >>>>>>>>> On Thu, 27 Jun 2019 at 10:46, emailtokirank@gmail.com
<
> >>>>>>>>> emailtokirank@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 2019/06/26 21:58:02, Patrik Kleindl <pkleindl@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>> Hi Kiran
> >>>>>>>>>>> You can use the RocksDBConfigSetter and
pass
> >>>>>>>>>>>
> >>>>>>>>>>> options.setMaxOpenFiles(100);
> >>>>>>>>>>>
> >>>>>>>>>>> to all RocksDBs for the Streams application
which limits
> >>> how
> >>>>> many are
> >>>>>>>>>>> kept open at the same time.
> >>>>>>>>>>>
> >>>>>>>>>>> best regards
> >>>>>>>>>>>
> >>>>>>>>>>> Patrik
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 26 Jun 2019 at 16:14, emailtokirank@gmail.com
<
> >>>>>>>>>>> emailtokirank@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We are using Kafka streams DSL APIs
for doing some
> >>> counter
> >>>>>>>> aggregations
> >>>>>>>>>>>> (running on OpenJDK 11.0.2). Our topology
has some 400
> >>> sub
> >>>>>>>> topologies
> >>>>>>>>>> & we
> >>>>>>>>>>>> are using 8 partitions in source topic.
When we start
> >>>>> pumping more
> >>>>>>>>>> load, we
> >>>>>>>>>>>> start getting RockDBException stating
"too many open
> >>> files".
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here are the stack trace samples:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> ------------------------------------------------------------------------------------------
> >>>>>>>>>>>> Caused by: org.rocksdb.RocksDBException:
while open a
> >>> file
> >>>>> for
> >>>>>>>> lock:
> >>>>>>>>>>>> PPPPPPPPPPP.1512000000/LOCK: Too many
open files
> >>>>>>>>>>>>       at org.rocksdb.RocksDB.open(Native
Method)
> >>>>>>>>>>>>       at org.rocksdb.RocksDB.open(RocksDB.java:235)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156)
> >>>>>>>>>>>>       ... 24 common frames omitted
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Caused by:
> >>>>> org.apache.kafka.streams.errors.ProcessorStateException:
> >>>>>>>>>> Error
> >>>>>>>>>>>> while executing flush from store XXXXXXXXXXX.1512000000
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:397)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:388)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.Segments.flush(Segments.java:163)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.flush(RocksDBSegmentedBytesStore.java:178)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:130)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.MeteredWindowStore.flush(MeteredWindowStore.java:177)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:217)
> >>>>>>>>>>>>       ... 10 more
> >>>>>>>>>>>> Caused by: org.rocksdb.RocksDBException:
While open a
> >>> file
> >>>>> for
> >>>>>>>>>> appending:
> >>>>>>>>>>>> YYYYYYYYYYYYY.1512000000/000007.dbtmp:
Too many open
> >>> files
> >>>>>>>>>>>>       at org.rocksdb.RocksDB.flush(Native
Method)
> >>>>>>>>>>>>       at org.rocksdb.RocksDB.flush(RocksDB.java:3401)
> >>>>>>>>>>>>       at org.rocksdb.RocksDB.flush(RocksDB.java:3361)
> >>>>>>>>>>>>       at
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:395)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> ------------------------------------------------------------------------------------------
> >>>>>>>>>>>>
> >>>>>>>>>>>> We tried increasing the open files limit
at OS level to
> >>> some
> >>>>> decent
> >>>>>>>>>>>> number.. but still no luck. Obviously
we don't want to
> >>> have
> >>>>>>>> boundless
> >>>>>>>>>> open
> >>>>>>>>>>>> files..
> >>>>>>>>>>>>
> >>>>>>>>>>>> We also tried to play with commit interval(
> >>>>>>>> kafka.commit.interval.ms)
> >>>>>>>>>> and
> >>>>>>>>>>>> cache size (kafka.cache.max.bytes.buffering)
.. but no
> >>> luck
> >>>>> there
> >>>>>>>>>> either.
> >>>>>>>>>>>>
> >>>>>>>>>>>> KAFKA-3904 talks about it.. but it was
resolved long
> >>> back..
> >>>>>>>>>>>>
> >>>>>>>>>>>> Any other config tuning we have to do?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Appreciate any help in this regard!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Kiran
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi Patrik/All,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for providing some valuable pointer!
> >>>>>>>>>>
> >>>>>>>>>> I did that & it doesn't seems to work.
> >>>>>>>>>>
> >>>>>>>>>> Here is how my custom config setter looks like:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> ----------------------------------------------------------------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> @Override
> >>>>>>>>>>         public void setConfig(final String storeName,
final
> >>>>> Options
> >>>>>>>>>> options, final Map<String, Object> configs)
{
> >>>>>>>>>>           // See #1 below.
> >>>>>>>>>>           BlockBasedTableConfig tableConfig
= new
> >>>>>>>>>> org.rocksdb.BlockBasedTableConfig();
> >>>>>>>>>>
> >>>>>>>>>>           tableConfig.setBlockCacheSize(16 *
1024 * 1024L);
> >>>>>>>>>>           // See #2 below.
> >>>>>>>>>>           tableConfig.setBlockSize(16 * 1024L);
> >>>>>>>>>>           // See #3 below.
> >>>>>>>>>>           tableConfig.setCacheIndexAndFilterBlocks(false);
> >>>>>>>>>>          //
> >>>>> tableConfig.setPinL0FilterAndIndexBlocksInCache(true);
> >>>>>>>>>>           options.setMaxOpenFiles(-1);
> >>>>>>>>>>           options.setTableFormatConfig(tableConfig);
> >>>>>>>>>>           // See #4 below.
> >>>>>>>>>>           options.setMaxWriteBufferNumber(2);
> >>>>>>>>>>         }
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> ----------------------------------------------------------------------------------------------------
> >>>>>>>>>> I tried many options with this:
> >>>>>>>>>> 1. tableConfig.setCacheIndexAndFilterBlocks(true);
 ----> as
> >>> per
> >>>>> docs (
> >>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>
> https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks
> >>>>>>>> )
> >>>>>>>>>> if we set to true, the max_open_files shouldn't
play a role.
> >>> But
> >>>>> I was
> >>>>>>>>>> still getting too many open files exception
from Rocksdb
> >>>>>>>>>>
> >>>>>>>>>> 2. tableConfig.setCacheIndexAndFilterBlocks(true);
> >>>>>>>>>> tableConfig.setPinL0FilterAndIndexBlocksInCache(true);
---->
> >>> no
> >>>>> luck;
> >>>>>>>> same
> >>>>>>>>>> exception
> >>>>>>>>>>
> >>>>>>>>>> 3. tableConfig.setCacheIndexAndFilterBlocks(false);
 and
> >>>>>>>>>> options.setMaxOpenFiles(50000);    ----->
 This also
> >>> resulted
> >>>>> with
> >>>>>>>> same
> >>>>>>>>>> exception.. with java process having ~24K open
files
> >>>>>>>>>>
> >>>>>>>>>> 4. tableConfig.setCacheIndexAndFilterBlocks(false);
 and
> >>>>>>>>>> options.setMaxOpenFiles(-1);    ----->  This
also resulted
> >>> with
> >>>>> same
> >>>>>>>>>> exception.. with java process having ~24K ope
files. As per
> >>> the
> >>>>> doc,
> >>>>>>>> if we
> >>>>>>>>>> set to -1, it means infinite and controlled
by underlying OS
> >>>>> limit.
> >>>>>>>>>>
> >>>>>>>>>> I am using MacOS Mojave (10.14.4) and OpenJDK
11.0.2.  At OS
> >>>>> level, I
> >>>>>>>> have
> >>>>>>>>>> bumped the open files limit to 1 million.
> >>>>>>>>>>
> >>>>>>>>>> $ ulimit -a
> >>>>>>>>>> core file size          (blocks, -c) 0
> >>>>>>>>>> data seg size           (kbytes, -d) unlimited
> >>>>>>>>>> file size               (blocks, -f) unlimited
> >>>>>>>>>> max locked memory       (kbytes, -l) unlimited
> >>>>>>>>>> max memory size         (kbytes, -m) unlimited
> >>>>>>>>>> open files                      (-n) 1000000
> >>>>>>>>>> pipe size            (512 bytes, -p) 1
> >>>>>>>>>> stack size              (kbytes, -s) 8192
> >>>>>>>>>> cpu time               (seconds, -t) unlimited
> >>>>>>>>>> max user processes              (-u) 1418
> >>>>>>>>>> virtual memory          (kbytes, -v) unlimited
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Am I missing some other config here?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Kiran
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Patrik,
> >>>>>>>>
> >>>>>>>> Thanks for quick response!
> >>>>>>>>
> >>>>>>>> I just checked the state dir. It has total of 3152 sub
folders
> >>> and
> >>>>> total
> >>>>>>>> of 15451 files (within sub folders..).
> >>>>>>>>
> >>>>>>>> With this what should be the number that I should use?
> >>>>>>>>
> >>>>>>>> How many instances of rocksdb will be used? Any formula
to
> >>> determine
> >>>>> that?
> >>>>>>>>
> >>>>>>>> Thanks again,
> >>>>>>>> Kiran
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>> Hi John/Patrik,
> >>>>>
> >>>>> Thanks for sharing some more insights!
> >>>>>
> >>>>> So we have ~3.2K state store directories (which means those many
> >>> rocksdb
> >>>>> instances.
> >>>>>
> >>>>> So when we tried with these config params:
> >>>>> ------------------
> >>>>> cache.index.and.filter.blocks=false
> >>>>> max.open.files=-1
> >>>>> block.cache.size=100* 1024 * 1024L
> >>>>> block.size=8*1024L
> >>>>> max.write.buffer.number=2
> >>>>> ------------------
> >>>>> We still got too many open files exception from rocksdb side. At
that
> >>> time
> >>>>> the total open files on the VM were ~46K. At OS level we have
> >>> increased the
> >>>>> max open files limit to 1 million.
> >>>>>
> >>>>> As per above config, "max.open.files=-1"  it means infinite open
> files
> >>> for
> >>>>> rocksdb and it's controlled by OS open file limit (which is 1 million
> >>> in
> >>>>> our case). We are not able to understand why rocksdb is not honouring
> >>> our
> >>>>> config params?
> >>>>>
> >>>>> P.S: we will not go with "max.open.files=-1" in production.. we
are
> >>> just
> >>>>> trying to understand how to overcome this exception by trying various
> >>>>> combinations of config params.
> >>>>>
> >>>>> Are we missing some other key rocksdb configs here?
> >>>>>
> >>>>> Thanks,
> >>>>> Kiran
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>> Hi Sophie,
> >>>
> >>> If it's honouring the config params, then why it should give too many
> open
> >>> files exception when we have set 1 million at OS level? When we check
> the
> >>> open files count, we hardly see them at ~42K.. and it never goes beyond
> >>> 50K. Why is it so? are there any other config params  which we have to
> tune?
> >>>
> >>> Thanks,
> >>> Kiran
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message