samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Pan <nickpa...@gmail.com>
Subject Re: ChangeLog Question for TTL rocksDB stores
Date Thu, 28 Jan 2016 19:51:31 GMT
Hi, David,

The "compaction" referred to together w/ TTL is referring to RocksDb's
compaction, not the Kafka-based changelog topic. Currently, TTL is not
applied to Kafka-based changelog topic. SAMZA-677 is opened for this.

-Yi

On Thu, Jan 28, 2016 at 11:36 AM, David Garcia <dgarcia@homeaway.com.invalid
> wrote:

> Ok, that makes sense.  I had assumed that the changelog was supported
> because the docs mention that TTL is enforced upon ³compaction² (I had
> assumed compaction of the DB changelog).  Which topic does the TTL policy
> listen for the compaction of (since compaction policies of topics can
> differ)?
>
> -David
>
> On 1/27/16, 8:46 PM, "Jacob Maes" <jacob.maes@gmail.com> wrote:
>
> >Here's my understanding. The others can correct me if I'm mistaken.
> >
> >Samza provides the changelog functionality by intercepting RocksDB "put"
> >and "delete" operations. However, TTL is managed by RocksDB internally and
> >there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems
> >that arise with TTL and change logging:
> >1. Samza doesn't know when an entry expires, so it can't delete the
> >expired
> >entry from the changelog.
> >2. The changelog currently has no concept of entry age/timestamp, so when
> >the changelog is restored, it's unknown whether some subset (or all) of
> >the
> >entries should be immediately expired.
> >
> >These issues aren't insurmountable, but they weren't pursued for the
> >initial implementation. Perhaps because there was a shortage of use cases
> >that needed both TTL and changelogging, but I'm not sure.
> >
> >-Jake
> >
> >On Wed, Jan 27, 2016 at 6:19 PM, David Garcia
> ><dgarcia@homeaway.com.invalid>
> >wrote:
> >
> >> So, I saw this very scary message:
> >>
> >>
> >> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based
> >> store, changelog is not supported for TTL based stores, use at your own
> >> discretion
> >>
> >>
> >>
> >>
> >> A few of questions:
> >>
> >> 1.) Does this mean that this store is NOT backed by the changelog?
> >>
> >> 2.) Provided that the store IS backed by a change log, do the TTL
> >> expirations commit removals from the changelog (I.e. Nulls)...presumably
> >> upon compaction
> >>
> >> 3.) Can I please get a bit more detail on how TTL affects a changelog
> >> store?
> >>
> >>
> >> -David
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message