kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Mittal <sjmit...@gmail.com>
Subject Re: Delete "kafka-logs"
Date Wed, 23 Nov 2016 06:20:58 GMT
try bin/kafka-server-start.sh ./config/server.properties

Also check how many segments have got created in logs dir and size of each.
May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.



On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan <eranga.h.n@gmail.com>
wrote:

> As you suggested I did these changes to the server.properties file on all
> nodes.
>
> log.retention.check.interval.ms=10000
> log.retention.ms=10000
> log.segment.bytes=10490000
>
> How I start each server is by,
>
> /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh
> -daemon
> /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/server.properties
>
> However, I still get about 4GB log files. Did I miss anything?
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <osanda.deshan@gmail.com>
> <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> erangaheshan>
>    <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal <sjmittal@gmail.com>
> wrote:
>
> > Check these in server.properties
> > ############################# Log Retention Policy
> > #############################
> >
> > # The following configurations control the disposal of log segments. The
> > policy can
> > # be set to delete segments after a period of time, or after a given size
> > has accumulated.
> > # A segment will be deleted whenever *either* of these criteria are met.
> > Deletion always happens
> > # from the end of the log.
> >
> > # The minimum age of a log file to be eligible for deletion
> > log.retention.hours=168
> >
> > # A size-based retention policy for logs. Segments are pruned from the
> log
> > as long as the remaining
> > # segments don't drop below log.retention.bytes.
> > #log.retention.bytes=1073741824
> >
> > # The maximum size of a log segment file. When this size is reached a new
> > log segment will be created.
> > log.segment.bytes=1073741824
> >
> > Try by setting hours as 1 and set segment bytes as say 10 MB. You should
> be
> > able to see multiple segments created of size 20 MB under you kafka logs
> > dir
> > say: tmp\kafka-logs\<topic>\*.log
> >
> > As the files age > 1 hour it will be marked for deletion and be deleted
> > next time delete thread runs. By default the delete thread check is 5
> > minutes.
> >
> > Also make sure when you start server you specify this property as the
> > argument.
> >
> > bin\windows\kafka-server-start.bat config\server.properties
> >
> >
> >
> > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan <eranga.h.n@gmail.com>
> > wrote:
> >
> > > This is the exact parameter I added to server.config file on all the
> > nodes.
> > >
> > > log.retention.ms=1000
> > >
> > > Thanks,
> > > Regards,
> > >
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> > > erangaheshan>
> > >    <https://www.linkedin.com/in/erangaheshan>
> > >
> > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan <eranga.h.n@gmail.com>
> > > wrote:
> > >
> > > > I have 4 node cluster and node1 has zookeeper running. I added those
> > > > parameters on each of my server.properties files in kafka nodes.
> > > >
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > > <https://www.facebook.com/erangaheshan>
> > > > <https://twitter.com/erangaheshan>
> > > > <https://www.linkedin.com/in/erangaheshan>
> > > >
> > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal <sjmittal@gmail.com>
> > > wrote:
> > > >
> > > >> Kafka logs are not kept based on data consumption. They are deleted
> > > after
> > > >> it reaches a particular size and retention time.
> > > >> Even if they are consumed multiple times in that duration, they are
> > > still
> > > >> kept.
> > > >>
> > > >> Where have you added those params and how are you starting your
> > > zookeeper
> > > >> and server.
> > > >>
> > > >>
> > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan <
> eranga.h.n@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Sorry for repeatedly asking the same question, but although I
> tried
> > > all
> > > >> > those parameters; my log file still at large about 2.5GB. Is
it
> due
> > to
> > > >> the
> > > >> > data inside my topic is flushed to disk? I meant that there might
> be
> > > >> data
> > > >> > inside topic which is not consumed yet or not been deleted even
> > after
> > > >> > consumed.
> > > >> >
> > > >> > Thanks,
> > > >> > Regards,
> > > >> >
> > > >> >
> > > >> > Eranga Heshan
> > > >> > *Undergraduate*
> > > >> > Computer Science & Engineering
> > > >> > University of Moratuwa
> > > >> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > >> > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > >> > <https://www.facebook.com/erangaheshan>   <https://twitter.com/
> > > >> > erangaheshan>
> > > >> >    <https://www.linkedin.com/in/erangaheshan>
> > > >> >
> > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal <
> sjmittal@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Set some value to log.retention.ms which is ms time to retain
> log
> > > and
> > > >> > also
> > > >> > > log.segment.bytes which is size of a single log file.
> > > >> > > delete.topic.enable is needed if you want to delete the
topic
> > > itself.
> > > >> > >
> > > >> > > Thanks
> > > >> > > Sachin
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan <
> > > eranga.h.n@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > I tried log.retention.bytes=1049000 which is about
10MB. But
> it
> > > did
> > > >> not
> > > >> > > > work. Is it because I had not given any value to
> > > >> delete.topic.enable?
> > > >> > > >
> > > >> > > >
> > > >> > > > Eranga Heshan
> > > >> > > > *Undergraduate*
> > > >> > > > Computer Science & Engineering
> > > >> > > > University of Moratuwa
> > > >> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > >> > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > >> > > > <https://www.facebook.com/erangaheshan>   <
> https://twitter.com/
> > > >> > > > erangaheshan>
> > > >> > > >    <https://www.linkedin.com/in/erangaheshan>
> > > >> > > >
> > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal <
> > > sjmittal@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Check
> > > >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs
> > > >> > > > > log.retention.bytes
> > > >> > > > > log.retention.ms
> > > >> > > > > log.segment.bytes
> > > >> > > > > delete.topic.enable
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan
<
> > > >> eranga.h.n@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi all,
> > > >> > > > > >
> > > >> > > > > > I want to keep the size of kafka-logs reduced
as much as
> > > >> possible.
> > > >> > To
> > > >> > > > do
> > > >> > > > > it
> > > >> > > > > > I need to set it to delete its unwanted logs
frequently.
> Is
> > > >> there a
> > > >> > > way
> > > >> > > > > to
> > > >> > > > > > do it?
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Regards,
> > > >> > > > > >
> > > >> > > > > > Eranga Heshan
> > > >> > > > > > *Undergraduate*
> > > >> > > > > > Computer Science & Engineering
> > > >> > > > > > University of Moratuwa
> > > >> > > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > >> > > > > > Email: eranga@wso2.com <osanda.deshan@gmail.com>
> > > >> > > > > > <https://www.facebook.com/erangaheshan>
  <
> > > https://twitter.com/
> > > >> > > > > > erangaheshan>
> > > >> > > > > >    <https://www.linkedin.com/in/erangaheshan>
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message