samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Garcia <dgar...@homeaway.com.INVALID>
Subject Re: ChangeLog Question for TTL rocksDB stores
Date Thu, 28 Jan 2016 19:36:21 GMT
Ok, that makes sense.  I had assumed that the changelog was supported
because the docs mention that TTL is enforced upon ³compaction² (I had
assumed compaction of the DB changelog).  Which topic does the TTL policy
listen for the compaction of (since compaction policies of topics can
differ)?

-David

On 1/27/16, 8:46 PM, "Jacob Maes" <jacob.maes@gmail.com> wrote:

>Here's my understanding. The others can correct me if I'm mistaken.
>
>Samza provides the changelog functionality by intercepting RocksDB "put"
>and "delete" operations. However, TTL is managed by RocksDB internally and
>there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems
>that arise with TTL and change logging:
>1. Samza doesn't know when an entry expires, so it can't delete the
>expired
>entry from the changelog.
>2. The changelog currently has no concept of entry age/timestamp, so when
>the changelog is restored, it's unknown whether some subset (or all) of
>the
>entries should be immediately expired.
>
>These issues aren't insurmountable, but they weren't pursued for the
>initial implementation. Perhaps because there was a shortage of use cases
>that needed both TTL and changelogging, but I'm not sure.
>
>-Jake
>
>On Wed, Jan 27, 2016 at 6:19 PM, David Garcia
><dgarcia@homeaway.com.invalid>
>wrote:
>
>> So, I saw this very scary message:
>>
>>
>> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based
>> store, changelog is not supported for TTL based stores, use at your own
>> discretion
>>
>>
>>
>>
>> A few of questions:
>>
>> 1.) Does this mean that this store is NOT backed by the changelog?
>>
>> 2.) Provided that the store IS backed by a change log, do the TTL
>> expirations commit removals from the changelog (I.e. Nulls)...presumably
>> upon compaction
>>
>> 3.) Can I please get a bit more detail on how TTL affects a changelog
>> store?
>>
>>
>> -David
>>


Mime
View raw message