chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: log rotation
Date Wed, 27 Jan 2010 21:27:00 GMT
I think I have a lapse in my understanding then regarding the sequence
number. I thought that was a pointer to the current offset in the file for
the data that has been read. It sounds like that's not necessarily the case.
If the sequence number continues to grow without reset after rotation, then
how would the adapter know where to start tailing a file if the agent was
bounced. The sequence number would be larger than the size of the file.


On Wed, Jan 27, 2010 at 1:11 PM, Ariel Rabkin <asrabkin@gmail.com> wrote:

> Chukwa *should* keep the sequence number growing monotonically when
> files rotate. There is some test coverage for this, but my hunch is
> that there are corner cases we haven't got right.
>
> On Wed, Jan 27, 2010 at 1:08 PM, Bill Graham <billgraham@gmail.com> wrote:
> > This thread reminded me of another related question I've been meaning to
> > ask. My implementation is similar to the one you describe Ari, where we
> tail
> > a single file indefinitely, but periodically it gets rotated by the Linux
> > logrotate script, which handles cleanup of old files (I don't think
> Chukwa
> > should handle this necessarily).
> >
> > My question is how does the agent handle the character offset marker when
> a
> > file it's tailing gets truncated during rotation? Does it just reset it
> back
> > to 0 when it sees that the file suddenly has no content?
> >
> >
> > On Wed, Jan 27, 2010 at 10:48 AM, Ariel Rabkin <asrabkin@gmail.com>
> wrote:
> >>
> >> Suppose you have a file, foo, with a FileTailingAdaptor watching it.
> >> After rotation, you'll have a foo.1 and then another foo.  The
> >> FileTailingAdaptor will look at the latter.
> >>
> >> After those files have been collected, it's safe to delete them.
> >> Right now, Chukwa doesn't do this itself. Should it?
> >>
> >> --Ari
> >>
> >> On Wed, Jan 27, 2010 at 10:18 AM, Corbin Hoenes <corbin@tynt.com>
> wrote:
> >> > I've noticed some chatter about log rotation and FileTailingAdaptor:
> >> >  http://issues.apache.org/jira/browse/CHUKWA-421
> >> >
> >> > Is there some kind of documentation on how it deals with say a
> directory
> >> > of files that get rotated.  At first glance it looks like it keeps an
> >> > adaptor open forever and there doesn't appear to be a way to know that
> a
> >> > file has been processed by the adaptor so we can't really move files
> out,
> >> > etc...  Right now we are piping apache access logs through rotatelogs
> and
> >> > then using chukwa to tail these files.  Is there a better way?
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Ari Rabkin asrabkin@gmail.com
> >> UC Berkeley Computer Science Department
> >
> >
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Mime
View raw message