chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: log rotation
Date Wed, 27 Jan 2010 21:11:04 GMT
Chukwa *should* keep the sequence number growing monotonically when
files rotate. There is some test coverage for this, but my hunch is
that there are corner cases we haven't got right.

On Wed, Jan 27, 2010 at 1:08 PM, Bill Graham <billgraham@gmail.com> wrote:
> This thread reminded me of another related question I've been meaning to
> ask. My implementation is similar to the one you describe Ari, where we tail
> a single file indefinitely, but periodically it gets rotated by the Linux
> logrotate script, which handles cleanup of old files (I don't think Chukwa
> should handle this necessarily).
>
> My question is how does the agent handle the character offset marker when a
> file it's tailing gets truncated during rotation? Does it just reset it back
> to 0 when it sees that the file suddenly has no content?
>
>
> On Wed, Jan 27, 2010 at 10:48 AM, Ariel Rabkin <asrabkin@gmail.com> wrote:
>>
>> Suppose you have a file, foo, with a FileTailingAdaptor watching it.
>> After rotation, you'll have a foo.1 and then another foo.  The
>> FileTailingAdaptor will look at the latter.
>>
>> After those files have been collected, it's safe to delete them.
>> Right now, Chukwa doesn't do this itself. Should it?
>>
>> --Ari
>>
>> On Wed, Jan 27, 2010 at 10:18 AM, Corbin Hoenes <corbin@tynt.com> wrote:
>> > I've noticed some chatter about log rotation and FileTailingAdaptor:
>> >  http://issues.apache.org/jira/browse/CHUKWA-421
>> >
>> > Is there some kind of documentation on how it deals with say a directory
>> > of files that get rotated.  At first glance it looks like it keeps an
>> > adaptor open forever and there doesn't appear to be a way to know that a
>> > file has been processed by the adaptor so we can't really move files out,
>> > etc...  Right now we are piping apache access logs through rotatelogs and
>> > then using chukwa to tail these files.  Is there a better way?
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Ari Rabkin asrabkin@gmail.com
>> UC Berkeley Computer Science Department
>
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message