ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Budnikov <a.budnikov.ign...@gmail.com>
Subject Re: Cache expiry policy not deleting records from disk(native persistence)
Date Mon, 16 Sep 2019 08:37:05 GMT
Hi Denis,

That's on the expiry policies page: 
https://apacheignite.readme.io/docs/expiry-policies

On 13.09.2019 19:46, Denis Magda wrote:
> Artem, thanks, could you please share a reference to the updated page? 
> Can't find anything here:
> https://apacheignite.readme.io/docs/evictions
>
> Shiva, I've restarted the discussion on the dev list, to get to the 
> bottom of this gap and how it can be addressed:
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>
> -
> Denis
>
>
> On Fri, Sep 13, 2019 at 7:05 AM Artem Budnikov 
> <a.budnikov.ignite@gmail.com <mailto:a.budnikov.ignite@gmail.com>> wrote:
>
>     Hi Denis,
>
>     I updated the page about eviction policies. Not freeing up space
>     on disk was not implemented for reasons explained in the dev-list
>     thread. I'll update the page once more if/when a solution is
>     implemented.
>
>     Artem
>
>     On 13.09.2019 00:34, Denis Magda wrote:
>>     Shiva,
>>
>>     Hopefully, someone from the dev community will pick this ticket
>>     up soon and solve the task. In the meantime, Artem, would you
>>     mind documenting this limitation referring to ticket 10862?
>>
>>     -
>>     Denis
>>
>>
>>     On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar
>>     <shivakumar.eng@gmail.com <mailto:shivakumar.eng@gmail.com>> wrote:
>>
>>         I have filed a bug
>>         https://issues.apache.org/jira/browse/IGNITE-12152 but this
>>         is same as https://issues.apache.org/jira/browse/IGNITE-10862
>>         Any idea on the timeline of these tickets?
>>         In the documentation
>>         https://apacheignite.readme.io/v2.7/docs/expiry-policies
>>         it says when native persistence is enabled "*expired entries
>>         are removed from both memory and disk tiers*" but in the disk
>>         it just mark the pages as unwanted pages and same disk space
>>         used by these unwanted pages will be used to store new pages
>>         but it will not remove unwanted pages from disk and so it
>>         will not release disk space used by these unwanted pages.
>>
>>         here is the developer's discussion link
>>         http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>>
>>
>>         On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar
>>         <shivakumar.eng@gmail.com <mailto:shivakumar.eng@gmail.com>>
>>         wrote:
>>
>>             Hi
>>             I have deployed ignite on kubernetes and configured two
>>             seperate persistent volume for WAL and persistence.
>>             The issue Iam facing is same as
>>             https://issues.apache.org/jira/browse/IGNITE-10862
>>
>>             Thanks
>>             Shiva
>>
>>             On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov,
>>             <aealexsandrov@gmail.com
>>             <mailto:aealexsandrov@gmail.com>> wrote:
>>
>>                 Hello,
>>
>>                 I guess that generated WAL will take this disk space.
>>                 Please read about WAL here:
>>
>>                 https://apacheignite.readme.io/docs/write-ahead-log
>>
>>                 Please provide the size of every folder under
>>                 /opt/ignite/persistence.
>>
>>                 BR,
>>                 Andrei
>>
>>                 9/6/2019 9:45 PM, Shiva Kumar пишет:
>>>                 Hi all,
>>>                 I have set cache expiry policy like this
>>>
>>>
>>>                  </property>
>>>                  <property name="cacheConfiguration">
>>>                           <list>
>>>                               <bean id="cache-template-bean"
>>>                 abstract="true"
>>>                 class="org.apache.ignite.configuration.CacheConfiguration">
>>>                                 <property name="name"
>>>                 value="templateEternal*"/>
>>>                                 <property name="cacheMode"
>>>                 value="PARTITIONED"/>
>>>                                 <property name="backups" value="1"/>
>>>                                 <property name="groupName"
>>>                 value="groupEternal"/>
>>>                                 <property name="expiryPolicyFactory">
>>>                                   <bean
>>>                 class="javax.cache.expiry.CreatedExpiryPolicy"
>>>                 factory-method="factoryOf">
>>>                 <constructor-arg>
>>>                                       <bean
>>>                 class="javax.cache.expiry.Duration">
>>>                 <constructor-arg value="MINUTES"/>
>>>                 <constructor-arg value="10"/>
>>>                                       </bean>
>>>                 </constructor-arg>
>>>                                   </bean>
>>>                                 </property>
>>>
>>>                               </bean>
>>>                           </list>
>>>                  </property>
>>>
>>>
>>>                 And batch inserting records to one of the table
>>>                 which is created with above cache template.
>>>                 Around 10 minutes, I ingested ~1.5GB of data and
>>>                 after 10 minutes records started reducing(expiring)
>>>                 when I monitored from sqlline.
>>>
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 248896
>>>                 --------------------------------
>>>                 1 row selected (0.86 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 222174
>>>                 --------------------------------
>>>                 1 row selected (0.313 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 118154
>>>                 --------------------------------
>>>                 1 row selected (0.15 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800>
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 76061
>>>                 --------------------------------
>>>                 1 row selected (0.106 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800>
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 41671
>>>                 --------------------------------
>>>                 1 row selected (0.063 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 18455
>>>                 --------------------------------
>>>                 1 row selected (0.037 seconds)
>>>                 0: jdbc:ignite:thin://192.168.*.*:10800> select
>>>                 count(ID) from DIMENSIONS;
>>>                 --------------------------------
>>>
>>>                 COUNT(ID)
>>>                 --------------------------------
>>>
>>>                 0
>>>                 --------------------------------
>>>                 1 row selected (0.014 seconds)
>>>
>>>
>>>                 But in the meantime, the disk space used by the
>>>                 persistence store was in the same usage level
>>>                 instead of decreasing.
>>>
>>>
>>>                 [ignite@ignite-cluster-ign-shiv-0 ignite]$ while
>>>                 true ; do df -h /opt/ignite/persistence/; sleep 1s; done
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>                 Filesystem Size Used Avail Use% Mounted on
>>>                 /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence
>>>
>>>
>>>
>>>                 This means that expiry policy not deleting records
>>>                 from the disk, but ignite document says when expiry
>>>                 policy is set and native persistence is enabled then
>>>                 it deletes records from disk as well.
>>>                 Am I missing some configuration?
>>>                 Any help is appreciated.
>>>
>>>                 Shiva
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message