samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Pan <nickpa...@gmail.com>
Subject Re: RocksDB TTL not working
Date Fri, 11 Mar 2016 21:44:48 GMT
Hi, Rohit,

The log for RocksDB should show whether the compact happened or not. You
can find the log in the container's runtime state-store directory here:
<container-runtime-dir-depending-on-installation-script>/state/<store-name>/<Partition_num>/LOG

My guess is that you might not have enough entries / writes to trigger the
compaction yet. You can reach out to RocksDB community for questions
regarding to when and how the compaction is triggered.

Best,

-Yi

On Fri, Mar 11, 2016 at 11:36 AM, Rohit Bansal <bansalrohit90@gmail.com>
wrote:

> I waited for a considerable timing, around 15-20 minutes. There were no
> logs of compaction though. How can I check if compaction occured or not ?
>
> On Sat, Mar 12, 2016 at 1:04 AM, Yi Pan <nickpan47@gmail.com> wrote:
>
> > Hi, Rohit,
> >
> > RocksDB TTL is best-effort:
> > https://github.com/facebook/rocksdb/wiki/Time-to-Live. And the expired
> > records are removed during compaction.
> >
> > I wonder in your test case whether the compaction has ever happened. How
> > long did you wait to see whether records are gone or not?
> >
> > -Yi
> >
> > On Thu, Mar 10, 2016 at 9:50 PM, Rohit Bansal <bansalrohit90@gmail.com>
> > wrote:
> >
> > > Navina,
> > >
> > > Please find my configurations file below -
> > >
> > > # Licensed to the Apache Software Foundation (ASF) under one
> > > # or more contributor license agreements.  See the NOTICE file
> > > # distributed with this work for additional information
> > > # regarding copyright ownership.  The ASF licenses this file
> > > # to you under the Apache License, Version 2.0 (the
> > > # "License"); you may not use this file except in compliance
> > > # with the License.  You may obtain a copy of the License at
> > > #
> > > #   http://www.apache.org/licenses/LICENSE-2.0
> > > #
> > > # Unless required by applicable law or agreed to in writing,
> > > # software distributed under the License is distributed on an
> > > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> > > # KIND, either express or implied.  See the License for the
> > > # specific language governing permissions and limitations
> > > # under the License.
> > >
> > > job.factory.class=org.apache.samza.job.local.ThreadJobFactory
> > > job.name=stream-poc
> > > job.coordinator.system=kafka
> > > job.coordinator.replication.factor=1
> > >
> > > task.class=samza.stream.poc.Consolidate
> > > task.inputs=stream.consolidate
> > > task.consumer.batch.size=100
> > > task.window.ms=20000
> > >
> > > # Serializers
> > >
> > >
> >
> serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory
> > >
> > >
> >
> serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory
> > >
> > > # Systems
> > >
> > >
> >
> systems.stream.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
> > > systems.stream.samza.key.serde=string
> > > systems.stream.samza.msg.serde=string
> > > systems.stream.consumer.zookeeper.connect=localhost:2181
> > > systems.stream.producer.bootstrap.servers=localhost:9092
> > >
> > > # Kafka System
> > >
> > >
> >
> systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
> > > systems.kafka.samza.msg.serde=string
> > > systems.kafka.samza.key.serde=string
> > > systems.kafka.consumer.zookeeper.connect=localhost:2181
> > > systems.kafka.producer.bootstrap.servers=localhost:9092
> > >
> > >
> > >
> >
> stores.rocksdb.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
> > > stores.rocksdb.key.serde=string
> > > stores.rocksdb.msg.serde=string
> > > stores.rocksdb.rocksdb.ttl.ms=2000
> > >
> > >
> > > *My test case :-*
> > >
> > > *Putting data in store (with TTL 2 seconds) and then getting data from
> > > store and printing in console in window() method which is executed in
> > > windowing operation in every 20 seconds. Data is printed in all
> windowing
> > > operation executions and is not getting deleted. It should have printed
> > > null.*
> > >
> > > *Thanks*
> > >
> > > On Fri, Mar 11, 2016 at 12:31 AM, Navina Ramesh <
> > > nramesh@linkedin.com.invalid> wrote:
> > >
> > > > Hi Rohit,
> > > > Are you running samza 0.10.0 or the latest from trunk?
> > > >
> > > > It will be helpful to paste your config here. Please do not attach it
> > to
> > > > the email because the mail server filters attachments.
> > > >
> > > > Thanks!
> > > > Navina
> > > >
> > > > On Thu, Mar 10, 2016 at 3:05 AM, Rohit Bansal <
> bansalrohit90@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > My messages are not getting deleted even after I have mentioned
> 2000
> > ms
> > > > as
> > > > > TTL in properties file. Do I need to add any other properties or
> make
> > > > > changes to any configuration for TTL to evict messages from db ?
I
> > can
> > > > > attach my task file and properties file if needed.
> > > > >
> > > > > --
> > > > >
> > > > > Regards
> > > > >
> > > > > Rohit Bansal
> > > > >
> > > > > +91-7838732062
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Navina R.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards
> > >
> > > Rohit Bansal
> > >
> > > +91-7838732062
> > >
> >
>
>
>
> --
>
> Regards
>
> Rohit Bansal
>
> +91-7838732062
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message