kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eranga Heshan <eranga....@gmail.com>
Subject Re: Delete "kafka-logs"
Date Wed, 23 Nov 2016 07:07:51 GMT
The reason I set retention time to 10 seconds is because I run kafka for
about 4 minutes each time. What I am running is a microbenchmark and I take
throughput and latency values for 4mins of runtime.

However when I run I get these printed on terminal.

[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/00000000000035778556.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,516] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/00000000000035550774.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/00000000000036690192.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/00000000000036918223.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,582] INFO Rolled new log segment for
'incrementStream1-1' in 0 ms. (kafka.log.Log)
[2016-11-23 12:21:54,797] INFO Rolled new log segment for
'incrementStream3-0' in 1 ms. (kafka.log.Log)
[2016-11-23 12:21:55,493] INFO Rolled new log segment for
'incrementStream1-1' in 1 ms. (kafka.log.Log)
[2016-11-23 12:21:55,787] INFO Rolled new log segment for
'incrementStream3-0' in 0 ms. (kafka.log.Log)
[2016-11-23 12:21:56,439] INFO Rolled new log segment for
'incrementStream1-1' in 0 ms. (kafka.log.Log)

When I go to kafka-logs directory I can see the segments of 10MB are
created with an index file. But after the retention time what happens is
that the log files change their names from "00000000000009117624.log" to
"00000000000009117624.log.deleted". And the index files change their names
from "00000000000009117624.index" to "00000000000009117624.index.deleted".
Nothing gets removed from the directory.

Is there any particular reason for this to happen?

Thanks,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <osanda.deshan@gmail.com>
<https://www.facebook.com/erangaheshan>   <https://twitter.com/erangaheshan>
   <https://www.linkedin.com/in/erangaheshan>

On Wed, Nov 23, 2016 at 11:50 AM, Sachin Mittal <sjmittal@gmail.com> wrote:

> try bin/kafka-server-start.sh ./config/server.properties
>
> Also check how many segments have got created in logs dir and size of each.
> May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.
>
>
>
> On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan <eranga.h.n@gmail.com>
> wrote:
>
> > As you suggested I did these changes to the server.properties file on all
> > nodes.
> >
> > log.retention.check.interval.ms=10000
> > log.retention.ms=10000
> > log.segment.bytes=10490000
> >
> > How I start each server is by,
> >
> > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-
> server-start.sh
> > -daemon
> > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/
> server.properties
> >
> > However, I still get about 4GB log files. Did I miss anything?
> >
> > Thanks,
> > Regards,
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> > erangaheshan>
> >    <https://www.linkedin.com/in/erangaheshan>
> >
> > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal <sjmittal@gmail.com>
> > wrote:
> >
> > > Check these in server.properties
> > > ############################# Log Retention Policy
> > > #############################
> > >
> > > # The following configurations control the disposal of log segments.
> The
> > > policy can
> > > # be set to delete segments after a period of time, or after a given
> size
> > > has accumulated.
> > > # A segment will be deleted whenever *either* of these criteria are
> met.
> > > Deletion always happens
> > > # from the end of the log.
> > >
> > > # The minimum age of a log file to be eligible for deletion
> > > log.retention.hours=168
> > >
> > > # A size-based retention policy for logs. Segments are pruned from the
> > log
> > > as long as the remaining
> > > # segments don't drop below log.retention.bytes.
> > > #log.retention.bytes=1073741824
> > >
> > > # The maximum size of a log segment file. When this size is reached a
> new
> > > log segment will be created.
> > > log.segment.bytes=1073741824
> > >
> > > Try by setting hours as 1 and set segment bytes as say 10 MB. You
> should
> > be
> > > able to see multiple segments created of size 20 MB under you kafka
> logs
> > > dir
> > > say: tmp\kafka-logs\<topic>\*.log
> > >
> > > As the files age > 1 hour it will be marked for deletion and be deleted
> > > next time delete thread runs. By default the delete thread check is 5
> > > minutes.
> > >
> > > Also make sure when you start server you specify this property as the
> > > argument.
> > >
> > > bin\windows\kafka-server-start.bat config\server.properties
> > >
> > >
> > >
> > > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan <eranga.h.n@gmail.com>
> > > wrote:
> > >
> > > > This is the exact parameter I added to server.config file on all the
> > > nodes.
> > > >
> > > > log.retention.ms=1000
> > > >
> > > > Thanks,
> > > > Regards,
> > > >
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> > > > erangaheshan>
> > > >    <https://www.linkedin.com/in/erangaheshan>
> > > >
> > > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan <eranga.h.n@gmail.com
> >
> > > > wrote:
> > > >
> > > > > I have 4 node cluster and node1 has zookeeper running. I added
> those
> > > > > parameters on each of my server.properties files in kafka nodes.
> > > > >
> > > > >
> > > > > Eranga Heshan
> > > > > *Undergraduate*
> > > > > Computer Science & Engineering
> > > > > University of Moratuwa
> > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > > <https://www.facebook.com/erangaheshan>
> > > > > <https://twitter.com/erangaheshan>
> > > > > <https://www.linkedin.com/in/erangaheshan>
> > > > >
> > > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal <sjmittal@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Kafka logs are not kept based on data consumption. They are
> deleted
> > > > after
> > > > >> it reaches a particular size and retention time.
> > > > >> Even if they are consumed multiple times in that duration, they
> are
> > > > still
> > > > >> kept.
> > > > >>
> > > > >> Where have you added those params and how are you starting your
> > > > zookeeper
> > > > >> and server.
> > > > >>
> > > > >>
> > > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan <
> > eranga.h.n@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Sorry for repeatedly asking the same question, but although
I
> > tried
> > > > all
> > > > >> > those parameters; my log file still at large about 2.5GB.
Is it
> > due
> > > to
> > > > >> the
> > > > >> > data inside my topic is flushed to disk? I meant that there
> might
> > be
> > > > >> data
> > > > >> > inside topic which is not consumed yet or not been deleted
even
> > > after
> > > > >> > consumed.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Regards,
> > > > >> >
> > > > >> >
> > > > >> > Eranga Heshan
> > > > >> > *Undergraduate*
> > > > >> > Computer Science & Engineering
> > > > >> > University of Moratuwa
> > > > >> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > >> > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > >> > <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> > > > >> > erangaheshan>
> > > > >> >    <https://www.linkedin.com/in/erangaheshan>
> > > > >> >
> > > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal <
> > sjmittal@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Set some value to log.retention.ms which is ms time
to retain
> > log
> > > > and
> > > > >> > also
> > > > >> > > log.segment.bytes which is size of a single log file.
> > > > >> > > delete.topic.enable is needed if you want to delete
the topic
> > > > itself.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > > Sachin
> > > > >> > >
> > > > >> > >
> > > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan <
> > > > eranga.h.n@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > I tried log.retention.bytes=1049000 which is about
10MB. But
> > it
> > > > did
> > > > >> not
> > > > >> > > > work. Is it because I had not given any value
to
> > > > >> delete.topic.enable?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Eranga Heshan
> > > > >> > > > *Undergraduate*
> > > > >> > > > Computer Science & Engineering
> > > > >> > > > University of Moratuwa
> > > > >> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > >> > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > >> > > > <https://www.facebook.com/erangaheshan>
  <
> > https://twitter.com/
> > > > >> > > > erangaheshan>
> > > > >> > > >    <https://www.linkedin.com/in/erangaheshan>
> > > > >> > > >
> > > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal
<
> > > > sjmittal@gmail.com>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > Check
> > > > >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs
> > > > >> > > > > log.retention.bytes
> > > > >> > > > > log.retention.ms
> > > > >> > > > > log.segment.bytes
> > > > >> > > > > delete.topic.enable
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan
<
> > > > >> eranga.h.n@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi all,
> > > > >> > > > > >
> > > > >> > > > > > I want to keep the size of kafka-logs
reduced as much as
> > > > >> possible.
> > > > >> > To
> > > > >> > > > do
> > > > >> > > > > it
> > > > >> > > > > > I need to set it to delete its unwanted
logs frequently.
> > Is
> > > > >> there a
> > > > >> > > way
> > > > >> > > > > to
> > > > >> > > > > > do it?
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Regards,
> > > > >> > > > > >
> > > > >> > > > > > Eranga Heshan
> > > > >> > > > > > *Undergraduate*
> > > > >> > > > > > Computer Science & Engineering
> > > > >> > > > > > University of Moratuwa
> > > > >> > > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > >> > > > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > >> > > > > > <https://www.facebook.com/erangaheshan>
  <
> > > > https://twitter.com/
> > > > >> > > > > > erangaheshan>
> > > > >> > > > > >    <https://www.linkedin.com/in/erangaheshan>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message