ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Trollip <ryanonthebe...@gmail.com>
Subject Re: Defrag?
Date Sat, 03 Jul 2021 14:10:28 GMT
A rebuild of the cash reduced the size of the data dramatically.
Apparently ignite is not doing anything to rebalance or clean up pages.
I can't see how anyone using ignite native seriously will not have this
problem.

I wonder if this impacts the indexing also? And could be part of the lousy
performance we are having with ignite native.


On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip <ryanonthebeach@gmail.com> wrote:

> Hey Ilya
>
> It's the data tables that keep growing not the WAL.
> We will try to rebuild the cache and see if that fixes the issue
>
> On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
> wrote:
>
>> Hello!
>>
>> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter,
>> any specific caches that are growing unbound?
>>
>> If letter, you can try creating a new cache, moving the relevant data to
>> this new cache, switch to using it, and then drop the old cache - should
>> reclaim the space.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ryanonthebeach@gmail.com>:
>>
>>> Is this why the native disk storage just keeps growing and does not
>>> reduce after we delete from ignite using SQL?
>>> We are up to 80GB on disk now on some instances. We implemented a custom
>>> archiving feature to move older data out of ignite cache to a PostgresSQL
>>> database but when we delete that data from ignite instance, the disk data
>>> size ignite is using stays the same, and then keeps growing, and
>>> growing....
>>>
>>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dmagda@apache.org> wrote:
>>>
>>>> Ignite fellows,
>>>>
>>>> I remember some of us worked on the persistence defragmentation
>>>> features. Has it been merged?
>>>>
>>>> @Valentin Kulichenko <valentin.kulichenko@gmail.com> probably you know
>>>> the latest state.
>>>>
>>>> -
>>>> Denis
>>>>
>>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> You can probably drop the entire cache and then re-populate it via
>>>>> loadCache(), etc.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ryanonthebeach@gmail.com>:
>>>>>
>>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>>> storage and caching more selectively as the performance degrades
when there
>>>>>> is a lot of write/delete activity or tables with large amounts of
rows.
>>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>>
>>>>>> Is there any easy way to rebuild the entire native database after
>>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>>
>>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I don't think there's anything ready to use, but "killing
>>>>>>> performance" from fragmentation is also not something reported
too often.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ryanonthebeach@gmail.com
>>>>>>> >:
>>>>>>>
>>>>>>>> We see continual very large growth to data with ignite native.
We
>>>>>>>> have a very chatty use case that's creating and deleting
stuff often. The
>>>>>>>> data on disk just keeps growing at an explosive rate. So
much so we ported
>>>>>>>> this to a DB to see the difference and the DB is much smaller.
I was
>>>>>>>> searching to see if someone has the same issue. This is also
killing
>>>>>>>> performance.
>>>>>>>>
>>>>>>>> Founds this:
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>>
>>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup
of
>>>>>>>> pages?
>>>>>>>>
>>>>>>>> Has anyone implemented a workaround to rebuild the cache
and
>>>>>>>> indexes say on a weekly basis to get it to behave reasonably?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message