Added: websites/staging/trafficserver/trunk/content/docs/v2/admin/intro.htm ============================================================================== --- websites/staging/trafficserver/trunk/content/docs/v2/admin/intro.htm (added) +++ websites/staging/trafficserver/trunk/content/docs/v2/admin/intro.htm Mon Dec 19 22:58:01 2011 @@ -0,0 +1,90 @@ + + + +Apache Traffic Server Administrator's Guide + + + +

Overview

+

Apache Traffic Server™ speeds Internet access, enhances website performance, and delivers unprecedented web hosting capabilities.

+

This chapter discusses the following topics:

+ +

What Is Apache Traffic Server?

+

Global data networking has become part of everyday life: internet users request billions of documents and terabytes of data, on a daily basis, to and from all parts of the world. Information is free, abundant, and accessible. Unfortunately, global data networking can also be a nightmare for IT professionals as they struggle with overloaded servers and congested networks. It can be challenging to consistently and reliably accommodate society’s growing data demands.

+

Traffic Server is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at the edge of the network. This brings content physically closer to end users, while enabling faster delivery and reduced bandwidth use. Traffic Server is designed to improve content delivery for enterprises, Internet service providers (ISPs), backbone providers, and large intranets by maximizing existing and available bandwidth.

+

Traffic Server Deployment Options

+

To best suit your needs, Traffic Server can be deployed in several ways:

+ +

The following sections provide a summary of these Traffic Server deployment options.

+

Traffic Server as a Web Proxy Cache

+

As a web proxy cache, Traffic Server receives user requests for web content as those requests travel to the destined web server (origin server). If Traffic Server contains the requested content, then it serves the content directly. If the requested content is not available from cache, then Traffic Server acts as a proxy: it obtains the content from the origin server on the user’s behalf and also keeps a copy to satisfy future requests.

+

Traffic Server provides explicit proxy caching, in which the user’s client software must be configured to send requests directly to Traffic Server. Explicit proxy caching is described in the Explicit Proxy Caching section.

+

Traffic Server as a Reverse Proxy

+

As a reverse proxy, Traffic Server is configured to be the origin server to which the user is trying to connect (typically, the origin server’s advertised hostname resolves to Traffic Server, which acts as the real origin server). The reverse proxy feature is also called server acceleration. Reverse proxy is described in more detail in Reverse Proxy and HTTP Redirects.

+

Traffic Server in a Cache Hierarchy

+

Traffic Server can participate in flexible cache hierarchies, in which Internet requests not fulfilled from one cache are routed to other regional caches, thereby leveraging the contents and proximity of nearby caches. In a hierarchy of proxy servers, Traffic Server can act either as a parent or a child cache to other Traffic Server systems or to similar caching products.

+

Traffic Server supports ICP (Internet Cache Protocol) peering. Hierarchical caching is described in more detail in Hierarchical Caching.

+

Traffic Server Components

+

Traffic Server consists of several components that work together to form a web proxy cache you can easily monitor and configure. These main components are described below.

+

The Traffic Server Cache

+

The Traffic Server cache consists of a high-speed object database called the object store. The object store indexes objects according to URLs and associated headers. Using sophisticated object management, the object store can cache alternate versions of the same object (perhaps in a different language or encoding type). It can also efficiently store very small and very large objects, thereby minimizing wasted space. When the cache is full, Traffic Server removes stale data to ensure that the most requested objects are readily available and fresh.

+

Traffic Server is designed to tolerate total disk failures on any of the cache disks. If the disk fails completely, then Traffic Server marks the entire disk as corrupt and continues to use remaining disks. If all of the cache disks fail, then Traffic Server switches to proxy-only mode. You can partition the cache to reserve a certain amount of disk space for storing data for specific protocols and origin servers. For more information about the cache, see Configuring the Cache.

+

The RAM Cache

+

Traffic Server maintains a small RAM cache that contains extremely popular objects. This RAM cache serves the most popular objects as fast as possible and reduces load on disks, especially during temporary traffic peaks. You can configure the RAM cache size to suit your needs; for detailed information, refer to Changing the Size of the RAM Cache.

+

The Host Database

+

The Traffic Server host database stores the domain name server (DNS) entries of origin servers to which Traffic Server connects to fulfill user requests. This information is used to adapt future protocol interactions and optimize performance. Along with other information, the host database tracks:

+ +

The DNS Resolver

+

Traffic Server includes a fast, asynchronous DNS resolver to streamline conversion of hostnames to IP addresses. Traffic Server implements the DNS resolver natively by directly issuing DNS command packets rather than relying on slower, conventional resolver libraries. Since many DNS queries can be issued in parallel and a fast DNS cache maintains popular bindings in memory, DNS traffic is reduced.

+

Traffic Server Processes

+

Traffic Server contains three processes that work together to serve Traffic Server requests and manage/control/monitor the health of the Traffic Server system. The three processes are described below:

+ +

The figure below illustrates the three Traffic Server processes.

+

+

 

+

Administration Tools

+

Traffic Server offers the following administration options:

+ +

Traffic Analysis Options

+

Traffic Server provides several options for network traffic analysis and monitoring:

+ +

Traffic analysis options are described in more detail in Monitoring Traffic; Traffic Server logging options are described in Working with Log Files.

+

Traffic Server Security Options

+

Traffic Server provides numerous options that enable you to establish secure communication between the Traffic Server system and other computers on the network. Using the security options, you can do the following:

+ +

Traffic Server security options are described in more detail in Security Options.

+ + \ No newline at end of file Propchange: websites/staging/trafficserver/trunk/content/docs/v2/admin/intro.htm ------------------------------------------------------------------------------ svn:executable = * Added: websites/staging/trafficserver/trunk/content/docs/v2/admin/leftnav.html ============================================================================== --- websites/staging/trafficserver/trunk/content/docs/v2/admin/leftnav.html (added) +++ websites/staging/trafficserver/trunk/content/docs/v2/admin/leftnav.html Mon Dec 19 22:58:01 2011 @@ -0,0 +1,19 @@ + \ No newline at end of file Propchange: websites/staging/trafficserver/trunk/content/docs/v2/admin/leftnav.html ------------------------------------------------------------------------------ svn:executable = * Added: websites/staging/trafficserver/trunk/content/docs/v2/admin/log.htm ============================================================================== --- websites/staging/trafficserver/trunk/content/docs/v2/admin/log.htm (added) +++ websites/staging/trafficserver/trunk/content/docs/v2/admin/log.htm Mon Dec 19 22:58:01 2011 @@ -0,0 +1,990 @@ + + + +Traffic Server Administrator's Guide + + + +

Working with Log Files

+

Traffic Server generates log files that contain information about every request it receives and every error it detects.

+

This chapter discusses the following topics:

+ +

Understanding Traffic Server Log Files

+

Traffic Server records information about every transaction (or request) it processes and every error it detects in log files. Traffic Server keeps three types of log files:

+ +

By default, Traffic Server creates both error and event log files and records system information in system log files. You can disable event logging and/or error logging by setting the configuration variable proxy.config.log.logging_enabled (in the records.config file) to one of the following values:

+

Understanding Event Log Files

+

Event log files record information about every request that Traffic Server processes. By analyzing the log files, you can determine how many people use the Traffic Server cache, how much information each person requested, what pages are most popular, and so on. Traffic Server supports several standard log file formats, such as Squid and Netscape, as well as user-defined custom formats. You can analyze the standard format log files with off-the-shelf analysis packages. To help with log file analysis, you can separate log files so they contain information specific to protocol or hosts. You can also configure Traffic Server to roll log files automatically at specific intervals during the day or when they reach a certain size.

+

The following sections describe the Traffic Server logging system features and discuss how to:

+ +

Managing Event Log Files

+

Traffic Server enables you to control where event log files are located and how much space they can consume. Additionally you can specify how to handle low disk space in the logging directory.

+

Choosing the Logging Directory

+

By default, Traffic Server writes all event log files in the logs directory located in the directory where you installed Traffic Server. To use a different directory, refer to Setting Log File Management Options.

+

Controlling Logging Space

+

Traffic Server enables you to control the amount of disk space that the logging directory can consume. This allows the system to operate smoothly within a specified space window for a long period of time. After you establish a space limit, Traffic Server continues to monitor the space in the logging directory. When the free space dwindles to the headroom limit (see Setting Log File Management Options), it enters a low space state and takes the following actions:

+ +

You can run a cron script in conjunction with Traffic Server to automatically remove old log files from the logging directory before Traffic Server enters the low space state. Relocate the old log files to a temporary partition, where you can run a variety of log analysis scripts. Following analysis, either compress the logs and move to an archive location, or simply delete them.

+

Setting Log File Management Options

+

To set log management options, follow the steps below:

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.logfile_dirSpecify the path to the directory in which you want to store event log files. This can be an absolute path or a path relative to the directory in which Traffic Server is installed. The default is logs located in the Traffic Server installation directory.
    + Note: The directory you specify must already exist.
    proxy.config.log.max_space_mb_for_logsEnter the maximum amount of space you want to allocate to the logging directory. The default value is 2000 MB.
    + Note: All files in the logging directory contribute to the space used, even if they are not log files.
    proxy.config.log.max_space_mb_headroomEnter the tolerance for the log space limit. The default value is 10 MB.
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

Choosing Event Log File Formats

+

Traffic Server supports the following log file formats:

+ +

In addition to the standard and custom log file format, you can choose whether to save log files in binary or ASCII; refer to Choosing Binary or ASCII.
+Event log files consume substantial disk space. Creating log entries in multiple formats at the same time can consume disk resources very quickly and adversely impact Traffic Server performance.

+

Using Standard Formats

+

The standard log formats include Squid, Netscape Common, Netscape extended, and Netscape Extended-2. The standard log file formats can be analyzed with a wide variety of off-the-shelf log-analysis packages. You should use one of the standard event log formats unless you need information that these formats do not provide. Refer to Using the Custom Format.

+

Setting Standard Log File Format Options

+

Set standard log file format options by following the steps below:

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. To use the Squid format, edit the following variables:
  4. + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.squid_log_enabledSet this variable to 1 to enable the Squid log file format.
    proxy.config.log.squid_log_is_asciiSet this variable to 1 to enable ASCII mode.
    Set this variable to 0 to enable binary mode.
    proxy.config.log.squid_log_nameEnter the name you want to use for Squid event log files. The default is squid.
    proxy.config.log.squid_log_headerEnter the header text you want to display at the top of the Squid log files. Enter NULL if you do not want to use a header.
    +
  5. To use the Netscape Common format, edit the following variables:
  6. + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.common_log_enabledSet this variable to 1 to enable the Netscape Common log file format.
    proxy.config.log.common_log_is_asciiSet this variable to 1 to enable ASCII mode.
    Set this variable to 0 to enable binary mode.
    proxy.config.log.common_log_nameEnter the name you want to use for Netscape Common event log files. The default is common.
    proxy.config.log.common_log_headerEnter the header text you want to display at the top of the Netscape Common log files. Enter NULL if you do not want to use a header.
    +
  7. To use the Netscape Extended format, edit the following variables:
  8. + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.extended_log_enabledSet this variable to 1 to enable the Netscape Extended log file format.
    proxy.config.log.extended_log_is_asciiSet this variable to 1 to enable ASCII mode.
    Set this variable to 0 to enable binary mode.
    proxy.config.log.extended_log_nameEnter the name you want to use for Netscape Extended event log files. The default is extended.
    proxy.config.log.extended_log_headerEnter the header text you want to display at the top of the Netscape Extended log files. Enter NULL if you do not want to use a header.
    +
  9. To use the Netscape Extended-2 format, edit the following variables:
  10. + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.extended2_log_enabledSet this variable to 1 to enable the Netscape Extended-2 log file format.
    proxy.config.log.extended2_log_is_asciiSet this variable to 1 to enable ASCII mode.
    Set this variable to 0 to enable binary mode.
    proxy.config.log.extended2_log_nameEnter the name you want to use for Netscape Extended-2 event log files. The default is extended2.
    proxy.config.log.extended2_log_headerEnter the header text you want to display at the top of the Netscape Extended-2 log files. Enter NULL if you do not want to use a header.
    +
  11. Save and close the records.config file.
  12. +
  13. Navigate to the Traffic Server bin directory.
  14. +
  15. Run the command traffic_line -x to apply the configuration changes.
  16. +
+

Using the Custom Format

+

The XML-based custom log format is more flexible then the standard log file formats and gives you more control over the type of information recorded in log files. You should create a custom log format if you need data for analysis that's not available in the standard formats. You can decide what information to record for each Traffic Server transaction and create filters that specify which transactions to log.

+

The heart of the XML-based custom logging feature is the XML-based logging configuration file (logs_xml.config) that enables you to create very modular descriptions of logging objects. The logs_xml.config file uses three types of objects to create custom log files, as detailed below. To generate a custom log format, you must specify at least one LogObject definition (one log file is produced for each LogObject definition).

+ +
To generate a custom log format:
+
    +
  1. In a text editor, open the records.config file located in the Traffic Server config directory.
  2. +
  3. Edit the following variables:
  4. + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.custom_logs_enabledSet this variable to 1 to enable custom logging.
    proxy.config.log.xml_logs_configMake sure this variable is set to 1 (the default value).
    +
  5. Save and close the records.config file.
  6. +
  7. Open the logs_xml.config file located in the Traffic Server config directory.
  8. +
  9. Add LogFormat, LogFilter, and LogObject specifications to the configuration file. For detailed information about this file, see logs_xml.config.
  10. +
  11. Save and close the logs_xml.config file.
  12. +
  13. Navigate to the Traffic Server bin directory.
  14. +
  15. Run the command traffic_line -x to apply your configuration changes.
  16. +
+

Creating Summary Log Files

+

Traffic Server performs several hundred operations per second; therefore, event log files can quickly grow to large sizes. Using SQL-like aggregate operators, you can configure Traffic Server to create summary log files that summarize a set of log entries over a specified period of time. This can significantly reduce the size of the log files generated.

+

To generate a summary log file, create a LogFormat object in the XML-based logging configuration file (logs_xml.config) using the SQL-like aggregate operators below. You can apply each of these operators to specific fields, over a specified interval.

+ +
To create a summary log file format:
+
    +
  1. Access the logs_xml.config file located in the Traffic Server config directory.
  2. +
  3. Define the format of the log file as follows:
    <LogFormat>
    <Name = "summary"/>
    <Format = "%<operator(field)> : %<operator(field)>"/>
    <Interval = "n"/>
    </Format>
    +
    + where operator is one of the five aggregate operators (COUNT, SUM, AVERAGE, FIRST, LAST), field is the logging field you want to aggregate, and n is the interval (in seconds) between summary log entries. You can specify more than one operator in the format line. For more information, refer to logs_xml.config.
    +
    + The following example format generates one entry every 10 seconds. Each entry contains the timestamp of the last entry of the interval, a count of the number of entries seen within that 10-second interval, and the sum of all bytes sent to the client: +
    <LogFormat>
    <Name = "summary"/>
    <Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>"/>
    <Interval = "10"/>
    </Format>

    +
    + IMPORTANT: You cannot create a format specification that contains both aggregate operators and regular fields. For example, the following specification would be invalid: +
    <Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)> : %<cqu>"/>
    +
  4. +
  5. Define a LogObject that uses this format.
  6. +
  7. Save your changes and close the logs_xml.config file. Run the command traffic_line -x from the Traffic Server bin directory to apply configuration changes .
  8. +
+

Choosing Binary or ASCII

+

You can configure the Traffic Server to create event log files in either of the following:

+ +

While binary log files typically require less disk space, there are exceptions.
+ For example: the value 0 (zero) requires only one byte to store in ASCII, but requires four bytes when stored as a binary integer. Conversely: if you define a custom format that logs IP addresses, then a binary log file would only require four bytes of storage per 32-bit address. However, the same IP address stored in dot notation would require around 15 characters (bytes) in an ASCII log file. Therefore, it's wise to consider the type of data that will be logged before you select ASCII or binary for your log files. For example, you might try logging for one day using ASCII and then another day using binary. If the number of requests is roughly the same for both days, then you can calculate a rough metric that compares the two formats.

+

For standard log formats, select Binary or ASCII (refer to Setting Standard Log File Format Options). For the custom log format, specify ASCII or Binary mode in the LogObject (refer to Using the Custom Format). In addition to the ASCII and binary options, you can also write custom log entries to a UNIX-named pipe (i.e., a buffer in memory). Other processes can then read the data using standard I/O functions. The advantage of using this option is that Traffic Server does not have to write to disk, which frees disk space and bandwidth for other tasks. In addition, writing to a pipe does not stop when logging space is exhausted because the pipe does not use disk space. Refer to logs_xml.config for more information about the ASCII_PIPE option.

+

Using logcat to Convert Binary Logs to ASCII

+

You must convert a binary log file to ASCII before you can analyze it using standard tools.

+
To convert a binary log file to ASCII:
+
    +
  1. Navigate to the directory that contains the binary log file.
  2. +
  3. Make sure that the logcat utility is in your path.
  4. +
  5. Enter the following command:
    logcat options input_filename...
    + The following table describes the command-line options.
    +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    OptionDescription
    -o output_fileSpecifies where the command output is directed.
    -aAutomatically generates the output filename based on the input filename. If the input is from stdin, then this option is ignored. For example:
    logcat -a squid-1.blog squid-2.blog squid-3.blog
    generates
    squid-1.log, squid-2.log, squid-3.log
    -SAttempts to transform the input to Squid format, if possible.
    -CAttempts to transform the input to Netscape Common format, if possible.
    -EAttempts to transform the input to Netscape Extended format, if possible.
    -2Attempt to transform the input to Netscape Extended-2 format, if possible.
    +
    + Note: Use only one of the following options at any given time: -S, -C, -E, or -2.
    + If no input files are specified, then logcat reads from the standard input (stdin). If you do not specify an output file, then logcat writes to the standard output (stdout).

    For example, to convert a binary log file to an ASCII file, you can use the logcat command with either of the following options below: +

    logcat binary_file > ascii_file
    logcat -o ascii_file binary_file
    + +The binary log file is not modified by this command.
+

Rolling Event Log Files

+

Traffic Server provides automatic log file rolling. This means that at specific intervals during the day or when log files reach a certain size, Traffic Server closes its current set of log files and opens new log files. You should roll log files several times a day. Rolling every six hours is a good guideline to start with.

+

Log file rolling offers the following benefits:

+ +

Rolled Log Filename Format

+

Traffic Server provides a consistent naming scheme for rolled log files that enables you to easily identify log files. When Traffic Server rolls a log file, it saves and closes the old file before it starts a new file. Traffic Server renames the old file to include the following information:

+ +

Timestamps have the following format:
%Y%M%D.%Hh%Mm%Ss-%Y%M%D.%Hh%Mm%Ss

+

The following table describes the format:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CodeDescription
%YThe year in four-digit format. For example: 2000.
%MThe month in two-digit format, from 01-12. For example: 07.
%DThe day in two-digit format, from 01-31. For example: 19.
%HThe hour in two-digit format, from 00-23. For example: 21.
%MThe minute in two-digit format, from 00-59. For example: 52.
%SThe second in two-digit format, from 00-59. For example: 36.
+
+

The following is an example of a rolled log filename:
squid.log.mymachine.20000912.12h00m00s-20000913.12h00m00s.old
+
+The logging system buffers log records before writing them to disk. When a log file is rolled, the log buffer might be partially full. If it is, then the first entry in the new log file will have a timestamp earlier than the time of rolling. When the new log file is rolled, its first timestamp will be a lower bound for the timestamp of the first entry.

+

For example, suppose logs are rolled every three hours, and the first rolled log file is:
+ squid.log.mymachine.19980912.12h00m00s-19980912.03h00m00s.old
+
+ If the lower bound for the first entry in the log buffer at 3:00:00 is 2:59:47, then the next log file will have the following timestamp when rolled: +
+ squid.log.mymachine.19980912.02h59m47s-19980912.06h00m00s.old

+

The contents of a log file are always between the two timestamps. Log files do not contain overlapping entries, even if successive timestamps appear to overlap.

+

Rolling Intervals

+

Log files are rolled at specific intervals relative to a given hour of the day. Two options control when log files are rolled:

+ +

Both the offset hour and the rolling interval determine when log file rolling starts. Rolling occurs every rolling interval and at the offset hour. For example, if the rolling interval is six hours and the offset hour is 0 (midnight), then the logs will roll at midnight (00:00), 06:00, 12:00, and 18:00 each day. If the rolling interval is 12 hours and the offset hour is 3, then logs will roll at 03:00 and 15:00 each day.

+

Setting Log File Rolling Options

+

To set log file rolling options and/or configure Traffic Server to roll log files when they reach a certain size, follow the steps below:

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. + + + + + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.rolling_enabledSet this variable to one of the following values:
    + 1 to enable log file rolling at specific intervals during the day.
    + 2 to enable log file rolling when log files reach a specific size.
    + 3 to enable log file rolling at specific intervals during the day or when log files reach a specific size (whichever occurs first).
    + 4 to enable log file rolling at specific intervals during the day when log files reach a specific size (at a specified time if the file is of the specified size).
    proxy.config.log.rolling_size_mbSpecifies the size that log files must reach before rolling takes place.
    proxy.config.log.rolling_offset_hrSet this variable to the specific time each day you want log file rolling to take place. Traffic Server forces the log file to be rolled at the offset hour each day.
    proxy.config.log.rolling_interval_secSet this variable to the rolling interval in seconds. The minimum value is 300 seconds (5 minutes). The maximum value is 86400 seconds (one day). Note: If you start Traffic Server within a few minutes of the next rolling time, then rolling might not occur until the next rolling time.
    proxy.config.log.auto_delete_rolled_fileSet this variable to 1 to enable autodeletion of rolled files.
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

You can fine-tune log file rolling settings for a custom log file in the LogObject specification in the logs_xml.config file. The custom log file uses the rolling settings in its LogObject, which override the default settings you specify in Traffic Manager or the records.config file described above.

+

Splitting Event Log Files

+

By default, Traffic Server uses standard log formats and generates log files that contain HTTP & ICP transactions in the same file. However, you can enable log splitting if you prefer to log transactions for different protocols in separate log files.

+

ICP Log Splitting

+

When ICP log splitting is enabled, Traffic Server records ICP transactions in a separate log file with a name that contains icp. For example: if you enable the Squid format, then all ICP transactions are recorded in the squid-icp.log file. When you disable ICP log splitting, Traffic Server records all ICP transactions in the same log file as HTTP transactions.

+

HTTP Host Log Splitting

+

HTTP host log splitting enables you to record HTTP transactions for different origin servers in separate log files. When HTTP host log splitting is enabled, Traffic Server creates a separate log file for each origin server that's listed in the log_hosts.config file. When both ICP and HTTP host log splitting are enabled, Traffic Server generates separate log files for HTTP transactions (based on the origin server) and places all ICP transactions in their own respective log files. For example, if the log_hosts.config file contains the two origin servers uni.edu and company.com and Squid format is enabled, then Traffic Server generates the following log files:

+
+ + + + + + + + + + + + + + + + + + + + + +
Log FilenameDescription
squid-uni.edu.logAll HTTP transactions for uni.edu
squid-company.com.logAll HTTP transactions for company.com
squid-icp.logAll ICP transactions for all hosts
squid.logAll HTTP transactions for other hosts
+
+

If you disable ICP log splitting, then ICP transactions are placed in the same log file as HTTP transactions. Using the hosts and log format from the previous example, Traffic Server generates the log files below:

+
+ + + + + + + + + + + + + + + + + +
Log FilenameDescription
squid-uni.edu.logAll entries for uni.edu
squid-company.com.logAll entries for company.com
squid.logAll other entries
+
+

Traffic Server also enables you to create XML-based Custom Log Formats that offer even greater control over log file generation.

+

Setting Log Splitting Options

+

To set log splitting options, follow the steps below:

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. +
    + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.separate_icp_logsSet this variable to 1 to record all ICP transactions in a separate log file.
    Set this variable to 0 to record all ICP transactions in the same log file as HTTP transactions.
    Set this variable to -1 to filter all ICP transactions from the standard log files.
    proxy.config.log.separate_host_logsSet this variable to 1 to record HTTP transactions for each host listed in log_hosts.config file in a separate log file.
    Set this variable to 0 to record all HTTP transactions (for each host listed in log_hosts.config) in the same log file.
    +
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

Editing the log_hosts.config File

+

The default log_hosts.config file is located in the Traffic Server config directory. To record HTTP transactions for different origin servers in separate log files, you must specify the hostname of each origin server on a separate line in the log_hosts.config file. For example, if you specify the keyword sports, then Traffic Server records all HTTP transactions from sports.yahoo.com and www.foxsports.com in a log file called squid-sports.log (if the Squid format is enabled).

+

Note: If Traffic Server is clustered and you enable log file collation, then you should use the same log_hosts.config file on every Traffic Server node in the cluster.

+
To edit the log_hosts.config file follow the steps below:
+
    +
  1. In a text editor, open the log_hosts.config file located in the Traffic Server config directory.
  2. +
  3. Enter the hostname of each origin server on a separate line in the file: for example,
    webserver1
    webserver2
    webserver3

    +
  4. +
  5. Save and close the log_hosts.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

+Collating Event Log Files

+

You can use the Traffic Server log file collation feature to collect all logged information in one place. Log collation enables you to analyze a set of Traffic Server clustered nodes as a whole (rather than as individual nodes) and to use a large disk that might only be located on one of the nodes in the cluster. Traffic Server collates log files by using one or more nodes as log collation servers and all remaining nodes as log collation clients. When a Traffic Server node generates a buffer of event log entries, it first determines if it is the collation server or a collation client. The collation server node writes all log buffers to its local disk, just as it would if log collation was not enabled. Log collation servers can be standalone or they can be part of a node running Traffic Server.

+

The collation client nodes prepare their log buffers for transfer across the network and send the buffers to the log collation server. When the log collation server receives a log buffer from a client, it writes it to its own log file as if it was generated locally. For a visual representation of this, see the figure below.

+

+
+

Log collation

+
+

If log clients cannot contact their log collation server, then they write their log buffers to their local disks, into orphan log files. Orphan log files require manual collation.

+

Note: Log collation can have an impact on network performance. Because all nodes are forwarding their log data buffers to the single collation server, a bottleneck can occur. In addition, collated log files contain timestamp information for each entry, but entries in the files do not appear in strict chronological order. You may want to sort collated log files before doing analysis.

+

To configure Traffic Server to collate event log files, you must perform the following tasks:

+ +

Configuring Traffic Server to Be a Collation Server

+

To configure a Traffic Server node to be a collation server, simply edit a configuration file via the steps below. If you modify the collation port or secret after connections between the collation server and collation clients have been established, then you must restart Traffic Server.

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. +
    + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.collation_modeSet this variable to 1 to set this Traffic Server node as a log collation server.
    proxy.config.log.collation_portSet this variable to specify the port number used for communication with collation clients. The default port number is 8085.
    proxy.config.log.collation_secretSet this variable to specify the password used to validate logging data and prevent the exchange of arbitrary information.
    All collation clients must use this same secret.
    +
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

Using a Standalone Collator

+

If you do not want the log collation server to be a Traffic Server node, then you can install and configure a standalone collator (SAC) that will dedicate more of its power to collecting, processing, and writing log files.

+
To install and configure a standalone collator:
+
    +
  1. Configure your Traffic Server nodes as log collation clients; refer to Configuring Traffic Server to Be a Collation Client.
  2. +
  3. Copy the sac binary from the Traffic Server bin directory to the machine serving as the standalone collator.
  4. +
  5. Create a directory called config in the directory that contains the sac binary.
  6. +
  7. Create a directory called internal in the config directory you created in Step 3 (above). This directory is used internally by the standalone collator to store lock files.
  8. +
  9. Copy the records.config file from a Traffic Server node configured to be a log collation client to the config directory you created in Step 3 on the standalone collator.
    + The records.config file contains the log collation secret and the port you specified when configuring Traffic Server nodes to be collation clients. The collation port and secret must be the same for all collation clients and servers.
  10. +
  11. In a text editor, open the records.config file on the standalone collator and edit the following variable:
  12. +
    + + + + + + + + + +
    VariableDescription
    proxy.config.log.logfile_dirSet this variable to specify the directory on which you want to store the log files. You can specify an absolute path to the directory or a path relative to the directory from which the sac binary is executed.
    Note: The directory must already exist on the machine serving as the standalone collator.
    +
    +
  13. Save and close the records.config file.
  14. +
  15. Enter the following command:
    sac -c config
  16. +
+

Configuring Traffic Server to Be a Collation Client

+

To configure a Traffic Server node to be a collation client, follow the steps below. If you modify the collation port or secret after connections between the collation clients and the collation server have been established, then you must restart Traffic Server.

+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.collation_modeSet this variable to 2 to configure this Traffic Server node to be a log collation client and send standard formatted log entries to the collation server.
    To send custom XML-based formatted log entries to the collation server, you must add a log object specification to the logs_xml.config file; refer to Using the Custom Format.
    proxy.config.log.collation_hostHostname of the collation server.
    proxy.config.log.collation_portThe port used for communication with the collation server. The default port number is 8085.
    proxy.config.log.collation_secretThe password used to validate logging data and prevent the exchange of arbitrary information.
    proxy.config.log.collation_host_taggedSet this variable to 1 if you want the hostname of the collation client that generated the log entry to be included in each entry.
    Set this variable to 0 if you do not want the hostname of the collation client that generated the log entry to be included in each entry.
    proxy.config.log.max_space_mb_for_orphan_logsSet this variable to specify the maximum amount of space (in megabytes) you want to allocate to the logging directory on the collation client for storing orphan log files. Orphan log files are created when the log collation server cannot be contacted. The default value is 25 MB.
    +
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

Collating Custom Event Log Files

+

If you use custom event log files, then you must edit the logs_xml.config file (in addition to configuring a collation server and collation clients).

+
To collate custom event log files:
+
    +
  1. On each collation client, open the logs_xml.config file in a text editor (located in the Traffic Server config directory).
  2. +
  3. Add the CollationHosts attribute to the LogObject specification, as shown below:
    <LogObject>
    <Format = "squid"/>
    <Filename = "squid"/>
    <CollationHosts="ipaddress:port"/>
    </LogObject>

    + where ipaddress is the hostname or IP address of the collation server to which all log entries (for this object) are forwarded, and port is the port number for communication between the collation server and collation clients.
    +
  4. +
  5. Save and close the logs_xml.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -L to restart Traffic Server on the local node or traffic_line -M to restart Traffic Server on all the nodes in a cluster.
  10. +
+

Viewing Logging Statistics

+

Traffic Server generates logging statistics that enable you to see the following information:

+ +

You can retrieve the statistics via the Traffic Line command-line interface; refer to Monitoring Traffic.

+

Viewing Log Files

+

You can view the system, event, and error log files Traffic Server creates. You can also delete a log file or copy it to your local systemif you have the correct user permissions. Traffic Server displays only one MB of information in the log file. If the log file you select to view is bigger than 1MB, then Traffic Server truncates the file and displays a warning message indicating that the file is too big.

+

Example Event Log File Entries

+

This section shows an example log file entry in each of the standard log formats supported by Traffic Server: Squid, Netscape Common, Netscape Extended, and Netscape Extended-2.

+

Squid Format

+

The following figure shows a sample log entry in a squid.log file.

+

+

The following table describes each field.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldSymbolDescription
1cqtqThe client request timestamp in Squid format; the time of the client request in seconds since January 1, 1970 UTC (with millisecond resolution).
2ttmsThe time Traffic Server spent processing the client request; the number of milliseconds between the time the client established the connection with Traffic Server and the time Traffic Server sent the last byte of the response back to the client.
3chiThe IP address of the client’s host machine.
4crc/psscThe cache result code; how the cache responded to the request: HIT, MISS, and so on. Cache result codes are described here.
The proxy response status code (the HTTP response status code from Traffic Server to client).
5psqlThe length of the Traffic Server response to the client in bytes, including headers and content.
6cqhmThe client request method: GET, POST, and so on.
7cqucThe client request canonical URL; blanks and other characters that might not be parsed by log analysis tools are replaced by escape sequences. The escape sequence is a percentage sign followed by the ASCII code number of the replaced character in hex.
8caunThe username of the authenticated client. A hyphen (-) means that no authentication was required.
9phr/pqsnThe proxy hierarchy route; the route Traffic Server used to retrieve the object.
+ The proxy request server name; the name of the server that fulfilled the request. If the request was a cache hit, then this field contains a hyphen (-).
10psctThe proxy response content type; the object content type taken from the Traffic Server response header.
+
+

Netscape Common

+

The following figure shows a sample log entry in a common.log file.

+

+

Netscape Extended

+

The following figure shows a sample log entry in an extended.log file.

+

+

Netscape Extended-2

+

The following figure shows a sample log entry in an extended2.log file. The following table describes each field.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldSymbolDescription
  Netscape Common
1chiThe IP address of the client’s host machine.
2 This hyphen (-) is always present in Netscape log entries.
3caunThe authenticated client username. A hyphen (-) means no authentication was required.
4cqtdThe date and time of the client request, enclosed in brackets.
5cqtxThe request line, enclosed in quotes.
6psscThe proxy response status code (HTTP reply code).
7psclThe length of the Traffic Server response to the client in bytes.
  Netscape Extended
8ssscThe origin server response status code.
9sshlThe server response transfer length; the body length in the origin server response to Traffic Server, in bytes.
10cqblThe client request transfer length; the body length in the client request to Traffic Server, in bytes.
11pqblThe proxy request transfer length; the body length in the Traffic Server request to the origin server.
12cqhlThe client request header length; the header length in the client request to Traffic Server.
13pshlThe proxy response header length; the header length in the Traffic Server response to the client.
14pqhlThe proxy request header length; the header length in Traffic Server request to the origin server.
15sshlThe server response header length; the header length in the origin server response to Traffic Server.
16ttsThe time Traffic Server spent processing the client request; the number of seconds between the time that the client established the connection with Traffic Server and the time that Traffic Server sent the last byte of the response back to the client.
  Netscape Extended2
17phrThe proxy hierarchy route; the route Traffic Server used to retrieve the object.
18cfscThe client finish status code: FIN if the client request completed successfully or INTR if the client request was interrupted.
19pfscThe proxy finish status code: FIN if the Traffic Server request to the origin server completed successfully or INTR if the request was interrupted.
20crcThe cache result code; how the Traffic Server cache responded to the request: HIT, MISS, and so on. Cache result codes are described here.
+

 

+

Support for Traditional Custom Logging

+

Traffic Server supports traditional custom logging in addition to the XML-based custom logging, which is more versatile and therefore recommended.

+

Traffic Server's format converter only converts traditional log configuration files named logs.config. If you are using a traditional log configuration file with a name other than logs.config, then you must convert the file yourself after installation; refer to Using cust_log_fmt_cnvrt. If you opt to use traditional custom logging instead of the more versatile XML-based custom logging, then you must enable the traditional custom logging option manually. Furthermore, if you want to configure Traffic Server as a collation client that sends log entries in traditional custom formats, then you must set collation options manually. Use the following procedures.

+

Enabling Traditional Custom Logging

+

To enable custom logging, you must edit a configuration file manually. To edit your existing traditional custom log formats, modify the logs.config file as before.

+
To enable traditional custom logging:
+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. +
    + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.custom_logs_enabled Set this variable to 1 to enable custom logging.
    proxy.config.log.xml_logs_configSet this variable to 0 to disable XML-based custom logging.
    +
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

To configure your Traffic Server node to be a collation client and send traditional custom log files to the collation server, use the following procedure.

+
To configure Traffic Server as a collation client:
+
    +
  1. In a text editor, open the records.config file located in the config directory.
  2. +
  3. Edit the following variables:
  4. +
    + + + + + + + + + + + + + + + + + + + + + + + + + +
    VariableDescription
    proxy.config.log.collation_modeSet this variable to 3 to configure this Traffic Server node to be a log collation client and send log entries in traditional custom formats to the collation server.
    Set this variable to 4 to configure this Traffic Server node to be a log collation client and send log entries in both standard formats (Squid, Netscape) and traditional custom formats to the collation server.
    proxy.config.log.collation_hostSpecify the hostname of the collation server.
    proxy.config.log.collation_portSpecify the port Traffic Server uses to communicate with the collation server. The default port number is 8085.
    proxy.config.log.collation_secretSpecify the password used to validate logging data and prevent exchange of arbitrary information.
    proxy.config.log.collation_host_taggedSet this variable to 1 if you want the hostname of the collation client that generated the log entry to be included in each entry.
    Set this variable to 0 if you do not want the hostname of the collation client that generated the log entry to be included in each entry.
    +
    +
  5. Save and close the records.config file.
  6. +
  7. Navigate to the Traffic Server bin directory.
  8. +
  9. Run the command traffic_line -x to apply the configuration changes.
  10. +
+

Using cust_log_fmt_cnvrt

+

The format converter cust_log_fmt_cnvrt converts your traditional custom log configuration file (logs.config) to an XML-based custom log configuration file (logs_xml.config). This enables you to use XML-based custom logging.

+
To run the format converter:
+
    +
  1. Navigate to the Traffic Server bin directory.
  2. +
  3. Enter the command cust_log_fmt_cnvrt and include the options you want to use.
    + The format of the command is
    cust_log_fmt_cnvrt [-o output_file | -a] [-hnVw] [input_file..]
    +
    + The following table describes the command-line options.
  4. +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    OptionDescription
    -o output_fileSpecifies the name of the output file; you can specify one output file only. If you specify multiple input files, then the converter combines the converted output from all files into a single output file.
    + This option and the -a option are mutually exclusive. If you want to create multiple output files from multiple input files, then you must use the -a option. If you do not specify an output file (using the -o or -a options), then output goes to stdout.
    -aGenerates one output file for each input file. The format converter automatically creates the name of the output file from the name of the input file by replacing .config at the end of the filename with _xml.config.
    + Note: If the source filename does not contain a .config extension, then the converter creates the new filename by appending _xml.config to the source filename.
    -hDisplays a description of the cust_log_fmt_cnvrt options.
    -nAnnotates the output file(s) with comments about the success or failure of the translation process for each of the input lines. This option produces a comment at the beginning of the output file(s) that describes errors the format converter encountered while converting the file. The comment includes line number, input line type (format, filter, or unknown), and either a success status or a description of the error encountered.
    -VDisplays the version of the format converter you are running.
    -wOverwrites existing output files without warning. If you do not specify the -w option, then the format converter does not overwrite existing output files. If you specify an output file that already exists, then the converter does not convert the input file.
    input fileSpecifies the name of the input file. If you do not specify an input filename, then the format converter takes the input from stdin.
    +
    +
+

Examples

+

The following example converts the file logs.config and sends the results to stdout:
+cust_log_fmt_cnvrt logs.config

+The following example converts a logs.config file into a logs_xml.config file and annotates the output file (logs_xml.config) with comments about the success or failure of the translation process. If a file named logs_xml.config already exists, then the format converter overwrites it.
cust_log_fmt_cnvrt -o logs_xml.config -n -w logs.config

The following example converts the files x.config, y.config, and z.config into three separate output files called x_xml.config, y_xml.config, and z_xml.config:
cust_log_fmt_cnvrt -a x.config y.config z.config
+

+ + \ No newline at end of file Propchange: websites/staging/trafficserver/trunk/content/docs/v2/admin/log.htm ------------------------------------------------------------------------------ svn:executable = *