flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Zagrebin <and...@ververica.com>
Subject Re: Migrating Existing TTL State to 1.8
Date Wed, 15 May 2019 14:53:25 GMT
Hi Ning,

If you have not activated non-incremental checkpointing then taking a
savepoint is the only way to trigger the full snapshot. In any case, it
will take time.

The incremental cleanup strategy is applicable only for heap state backend
and does nothing for RocksDB backend. At the moment, you can combine only
compaction filter with full snapshotting cleanup with RocksDB backend.

Best,
Andrey

On Fri, Mar 15, 2019 at 11:56 PM Ning Shi <ningshi2@gmail.com> wrote:

> Hi Stefan,
>
> Thank you for the confirmation.
>
> Doing a one time cleanup with full snapshot and upgrading to Flink 1.8
> could work. However, in our case, the state is quite large (TBs).
> Taking a savepoint takes over an hour, during which we have to pause
> the job or it may process more events.
>
> The JavaDoc of `cleanupFullSnapshot` [1] says "Cleanup expired state
> in full snapshot on checkpoint.". My understanding is that the only
> way to take a full snapshot with RocksDB backend is to take a
> savepoint. Is there another way to take a full checkpoint?
>
> I noticed that Flink 1.8 also added an incremental cleanup strategy
> [2] by iterating through several keys at a time for each state access.
> If I combine this with the new compaction filter cleanup strategy,
> will it eventually remove all expired state without taking a full
> snapshot for upgrade?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/state/StateTtlConfig.Builder.html#cleanupFullSnapshot--
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/state/StateTtlConfig.Builder.html#cleanupIncrementally-int-boolean-
>
> Thanks,
>
> Ning
>
>
> On Wed, Mar 13, 2019 at 11:22 AM Stefan Richter <s.richter@ververica.com>
> wrote:
> >
> > Hi,
> >
> > If you are worried about old state, you can combine the compaction
> filter based TTL with other cleanup strategies (see docs). For example,
> setting `cleanupFullSnapshot` when you take a savepoint it will be cleared
> of any expired state and you can then use it to bring it into Flink 1.8.
> >
> > Best,
> > Stefan
>

Mime
View raw message