nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Burkey <>
Subject Re: LogFormatter
Date Tue, 03 Jan 2006 14:22:14 GMT
H Daniel,

    I agree with you. Using the LogFormatter is a bad design and is 
causing us problems as well. We use Spring and set the error handler to 
any objects that want it.

Daniel Feinstein wrote:
> Hi,
> This is my first email to the community. In our ( project 
> we integrated multiple lucene indexes and a nutch package. We use 
> the packages for about two years. Lucene is working great and we could 
> use it as a "black box". Unfortunately, we had to modify Nutch package 
> for our usage because of some leaks and a design problem in 
> LogFormatter class. This fact force us to use a customized version of 
> Nutch and avoid to update versions. At the last week we decided to 
> move to the latest version of Nutch but discovered that the 
> LogFormatter still have the same problem (leaks I believe has been fixed).
> The problem:
> In Nutch project LogFormatter has duplicated functionality:
> 1) Logger records format and
> 2) Severe error handler
> The first usage is standard and usually could be overwritten by a user 
> of the package by modifying file.
> The second usage is much more problematic because it affects the 
> behavior of the whole application (not only Nutch package). To support 
> the error handling LogFormatter enforce usage of the formatter class 
> by all classes of the whole application which uses Nutch package. This 
> is done by overwriting all the system handlers (class 
> java.util.logging.Handler). This operation prevents the application to 
> use its own log formatter. Also this cause 
> LogFormatter.hasLoggedSevere() to be sensitive to all severe records 
> in the big system but not only to relevant. More than that this flag, 
> LogFormatter.loggedSevere is never cleaned what means if an 
> application had one, even unrelated severe record, tools like Fetcher 
> will never run until the application will be restarted.
> I would like to suggest the following solutions:
> 1) To separate the functionality of log formatting and error handling or
> 2) Change LogFormatter class to be affected only by nutch package 
> functions
> For my opinion the first solution is much better especially if error 
> handling will be encapsulated for each task. I have found the 
> following usages of LogFormatter.hasLoggedSevere():
> - Fetcher
> - URLFilterChecker
> - ParseSegment
> Unfortunately I'm not familiar enough with the usages above to 
> implement this solution that why I suggest the second one. The 
> attached file is our implementation of LogFormatter class which we use 
> for more than a year. I hope this change will be accepted by the 
> community.
> Daniel
> _____________________
> */Daniel Feinstein/*
> Lead Software Engineer
> <>
> __________________________
> This email may contain confidential and privileged material for the 
> sole use of the intended recipient. Any review or distribution by 
> others is strictly prohibited. If you are not the intended recipient 
> please contact the sender and delete all copies.

Christopher Burkey

View raw message