james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noel J. Bergman" <n...@devtech.com>
Subject RE: Message path tracing
Date Tue, 29 Jul 2003 20:33:59 GMT
Serge Knystautas wrote:

> One issue I've noticed on server-user@ is people not knowing how their
> messages get routed.

> you could turn on a switch in the conf file and
> have per-message ID log files.

> The logging cycle (per file/message) would go like this...
>   - mark that it was received via smtp (or maybe fetchpop?)
>   - starting this processor
>   - any log messages from matchers/mailets also get put in here
>   - report what mailets do fire (?), so you can see which matchers
>     evaluate to true, and if a message keeps circulating.
>   - say it ended.

> if you do MailetContext.sendMail, that gets a new #, and the original
> message log file should report that it sent something and created that
> new #.  This should let you see.... 4.log has entry "sent message to
> create #5", see 5.log has entry "sent message to create #6", etc...

Lindsay Smith concured:
> That is a great idea.  The DEBUG logs I know have everything I need to
> analyse my mailets, but I find them over whelming to read.  I was just
> introduced to Log4J's LogFactor5 and I was pondering if that could help me
> with reading James logs, by somehow filtering, to group messages by email
> id, from the smtp log.

My first inclination is that Serge has the right idea, but possibly the
wrong solution to the problem.  Having per-message logs (Msg#.log) won't
scale, and I believe that problems arise other than in someone's controlled
test environment as they develop their pipeline logic.

Dynamic reconfiguration addresses some things, but (a) we should have a
general mechanism for  it, not an ad hoc approach for this one case, and (b)
doesn't address the problem where it the pipeline question isn't easily
reproduced under "laboratory conditions."

It seems to me, and there is also a bugzilla entry that relates to this,
that we should audit all of our message related log formats to make sure
that the content is parsable and has all of the necessary information to
associate log messages with mail messages, and then provide a log analyzer.

A hypothetical pipeline analyzer could merge the various existing logs, sort
them by message key, and then filter out events by the message(s) we're
interested in following.

Serge has made a point that logs for smtp traffic analysis and logs for
pipeline debugging might have different content/formats.  Possibly so, but I
don't believe that precludes using a log analyzer.

Either way, we have to go through SMTP handler, the pipeline, into matchers
and mailets (at least in my opinion), and the POP3 handler, and make some
logging changes.  At the minimum, it would seem to me that want to log

   - message creation/duplication
   - each match/service invocation
   - the state after each match/service

Each entry should have, at a minimum:

   - timestamp
   - message key
   - processor
   - log text

We currently have log entries like:

 <timestamp> INFO  smtpserver: Successfully spooled mail from <sender> for
 <timestamp> INFO  spoolmanager: ==== Begin processing mail <key> ====
 <timestamp> INFO  spoolmanager: Processing <key> through <processor>
 <timestamp> INFO  spoolmanager: ==== Removed from spool mail <key> ====
 <timestamp> DEBUG spoolmanager.<processor>: Checking <key> with <matcher>
 <timestamp> DEBUG spoolmanager.<processor>: Servicing <key> by <mailet>
 <timestamp> INFO  James.Mailet: RemoteDelivery: Attempting delivery of
<key-to-domain> to host <host> to addresses [<recipient-list>]
 <timestamp> INFO  James.Mailet: RemoteDelivery: Mail (<key-to-domain>) sent
successfully to <host>.
 <timestamp> INFO  James.Mailet: ToRepository: Storing mail <key> in
 <timestamp> INFO  James.Mailet: SenderInFakeDomain: No MX, A, or CNAME
record found for domain: <domain>
 <timestamp> INFO  James.Mailet: ToProcessor: Sending mail <instance> to

So you can see how we do/don't match up.  Multi-line (e.g., stack trace)
messages would also need to be handled.

I believe that Serge's idea is that we would instrument the SMTP handler
(probably also any similar message source, such as FetchMail), the spooler,
and the log operation(s) in the MailetContext.  We would have some redundant
logging code, but hopefully could keep it to a minimum.  I'm not sure how
this per-message log would tie into the Avalon code, but that's TBD if we go
this route.

Personally, I prefer the log analyzer approach for a few reasons:

  - logs for the log analyzer could be left on with
    some degree of verbosity
  - the per-message log mechanism would rapidly create
    too many directory entries for many platforms.
  - no redundant code
  - resolves bugzilla


	--- Noel

To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

View raw message