httpd-wiki-changes mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Httpd Wiki] Update of "ReInflating" by wrowe
Date Fri, 28 Sep 2007 09:55:53 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Httpd Wiki" for change notification.

The following page has been changed by wrowe:
http://wiki.apache.org/httpd/ReInflating

The comment on the change is:
Color commentary and the play-by-play

------------------------------------------------------------------------------
+ So you want to handle the text from a proxy or other app which has deflate'd the content?
- #FORMAT plain
- I've lost decorative text once already, explanations to follow
  
-   LogFormat "%h \"%r\" %>s %b (%O) TE:\"%{Transfer-Encoding}o\" CE:\"%{Content-Encoding}o\"
<-- %I AE:\"%{Accept-Encoding}i\" TE:\"%{Transfer-Encoding}i\" CE:\"%{Content-Encoding}i\""
compression
+ The first most valuable thing for diagnosing any issues is to create a compression log
+ which will show you the Accept-Encoding (AE) permitted by the client (browser) along
+ with the Transfer-Encoding (TE) and Content-Encoding (CE) for the request body with the
+ total input byte count.  We'll prefix each entry with the client ip/host and their request
+ line.  And the meat of the response to the client are the response size (bytes transmitted),
+ response Transfer-Encoding and response Content-Encoding.  Throughout these examples we'll
+ be sharing the results as gathered with the following config snippet;
  
-   CustomLog logs/compression.log compression
+ {{{
+ LogFormat "%h \"%r\" %>s %b (%O) TE:\"%{Transfer-Encoding}o\" CE:\"%{Content-Encoding}o\"
<-- %I AE:\"%{Accept-Encoding}i\" TE:\"%{Transfer-Encoding}i\" CE:\"%{Content-Encoding}i\""
compression
  
+ CustomLog logs/compression.log compression
+ }}}
  
+ The first thing is to provide a server to test against which deflates it's output.  We can
+ do this in the same instance of Apache, simply set up an alias to the htdocs for /deflated/
+ requests for the very same documents in /.
-   Alias /deflated "/path-to/htdocs"
-   <Location "/deflated">
-       SetOutputFilter DEFLATE
-   </Location>
  
-   127.0.0.1 "GET /LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 412 AE:"gzip,deflate"
TE:"-" CE:"-" 
-   127.0.0.1 "GET /local/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
418 AE:"gzip,deflate" TE:"-" CE:"-" 
+ {{{
+ Alias /deflated "/path-to/htdocs"
+ <Location "/deflated">
+     SetOutputFilter DEFLATE
+ </Location>
+ }}}
  
+ Requesting the same LICENSE.txt from both / and /deflated/ reveals the impact deflate can
offer;
  
-   ProxyPass /proxy/ http://127.0.0.1/
-   ProxyPassReverse /proxy/ http://127.0.0.1/
+ {{{
+ 127.0.0.1 "GET /LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 412 AE:"gzip,deflate"
TE:"-" CE:"-" 
+ 127.0.0.1 "GET /local/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <-- 418
AE:"gzip,deflate" TE:"-" CE:"-" 
+ }}}
  
+ Note the response body has shrunk nearly to a quarter of it's original size.
-   127.0.0.1 "GET /LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 501 AE:"gzip,deflate"
TE:"-" CE:"-" 
-   127.0.0.1 "GET /proxy/LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 418
AE:"gzip,deflate" TE:"-" CE:"-" 
-   127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
510 AE:"gzip,deflate" TE:"-" CE:"-" 
-   127.0.0.1 "GET /proxy/deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip"
<-- 427 AE:"gzip,deflate" TE:"-" CE:"-" 
  
+ For our first experiment (and we will keep it in our config as the LAST snippet,
+ because this would override any other example below if it appears first), we will
+ create a reverse-proxy (gateway) to our own server's content, everything requested
+ from /proxy/ will actually be proxied from / of our server.
  
+ {{{
-   ProxyPass /proxy/nodeflate/ http://127.0.0.1/deflated/
+ ProxyPass /proxy/ http://127.0.0.1/
+ ProxyPassReverse /proxy/ http://127.0.0.1/
+ }}}
-   <Location "/proxy/nodeflate/">
-       RequestHeader unset Accept-Encoding
-   </Location>
  
-   127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 39858 (40169) TE:"-" CE:"-" <--
479 AE:"-" TE:"-" CE:"-" 
-   127.0.0.1 "GET /proxy/nodeflate/LICENSE.txt HTTP/1.1" 200 39858 (40169) TE:"-" CE:"-"
<-- 428 AE:"-" TE:"-" CE:"-" 
+ The compression.log gets interesting here.  The first request to be satisfied is our
+ back-end proxied request from ourself, once it's completed the /proxy/ request can
+ be completed.  In this case, the responses are identical, we aren't manipulating
+ the Content-Encoding as the request passes through the proxy;
  
+ {{{
+ 127.0.0.1 "GET /LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 501 AE:"gzip,deflate"
TE:"-" CE:"-" 
+ 127.0.0.1 "GET /proxy/LICENSE.txt HTTP/1.1" 200 39858 (40146) TE:"-" CE:"-" <-- 418 AE:"gzip,deflate"
TE:"-" CE:"-" 
  
+ 127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
510 AE:"gzip,deflate" TE:"-" CE:"-" 
+ 127.0.0.1 "GET /proxy/deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip"
<-- 427 AE:"gzip,deflate" TE:"-" CE:"-" 
+ }}}
+ 
+ Now the quick method of disabling compression is to never request a compressed
+ document from the back-end machine, which we can accomplish by unset'ting the
+ Accept-Encoding header from the client.
+ 
+ {{{
+ ProxyPass /proxy/nodeflate/ http://127.0.0.1/deflated/
+ <Location "/proxy/nodeflate/">
+     RequestHeader unset Accept-Encoding
+ </Location>
+ }}}
+ 
+ And lo, neither the backend or proxy request appears to support gzip encoding;
+ 
+ {{{
+ 127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 39858 (40169) TE:"-" CE:"-" <-- 479
AE:"-" TE:"-" CE:"-" 
+ 127.0.0.1 "GET /proxy/nodeflate/LICENSE.txt HTTP/1.1" 200 39858 (40169) TE:"-" CE:"-" <--
428 AE:"-" TE:"-" CE:"-" 
+ }}}
+ 
+ But if we want to continue to accept deflated content from the back end server
+ for bandwidth conservation, it's possible to use mod_deflate to perform response
+ re-inflation.  We only want to do this if the back end server provided a response
+ in Content-Encoding:gzip, or else we will be trying to unpack an apparently corrupt
+ data stream;
+ 
+ {{{
    FilterProvider gzinflate INFLATE resp=Content-Encoding $gzip
  
    ProxyPass /proxy/reinflate/ http://127.0.0.1/deflated/
    <Location "/proxy/reinflate/">
        FilterChain gzinflate
    </Location>
+ }}}
  
-   127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
510 AE:"gzip,deflate" TE:"-" CE:"-" 
-   127.0.0.1 "GET /proxy/reinflate/LICENSE.txt HTTP/1.1" 200 39858 (40219) TE:"chunked" CE:"-"
<-- 428 AE:"gzip,deflate" TE:"-" CE:"-" 
+ Now when a request is made through /proxy/reinflate/, we will take the deflated
+ back end content and explode it.  Note that the (2nd) response to the client is
+ nearly four times larger, again, than the backend request;
  
+ {{{
+ 127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
510 AE:"gzip,deflate" TE:"-" CE:"-" 
+ 127.0.0.1 "GET /proxy/reinflate/LICENSE.txt HTTP/1.1" 200 39858 (40219) TE:"chunked" CE:"-"
<-- 428 AE:"gzip,deflate" TE:"-" CE:"-" 
+ }}}
  
-   FilterProvider ucase CaseFilter Content-Type $text/
+ Let's take this one step further, and add on a transformation.  There's an experimental
+ filter mod_case_filter which upper-cases the response, and it's perfect for illustrating

+ the transformation.  We can't upper-case the deflated binary stream, so we'll have to
+ inflate it, then transform it.
  
-   ProxyPass /proxy/ucase/ http://127.0.0.1/deflated/
-   <Location "/proxy/ucase/">
-       FilterChain +gzinflate +ucase
-   </Location>
- 
-   127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
513 AE:"gzip,deflate" TE:"-" CE:"-"
-   127.0.0.1 "GET /proxy/ucase/LICENSE.txt HTTP/1.1" 200 39858 (40219) TE:"chunked" CE:"-"
<-- 427 AE:"gzip,deflate" TE:"-" CE:"-"
- 
- ** NOTE One line bug in mod_case_filter.c that must be fixed in 2.2.6,
+ ** NOTE A one line bug in mod_case_filter.c that must be fixed in 2.2.6! Insert the line;
- insert the line;
+ {{{
          apr_brigade_cleanup(pbbIn);
+ }}}
  immediately before this line;
+ {{{
          return ap_pass_brigade(f->next, pbbOut);
+ }}}
  and rebuild the module.
  
+ So using the example above, we'll build on it with another FilterProvider we'll name ucase,
+ and apply both gzinflate and ucase in that order to all /proxy/ucase/ requests;
  
-   FilterProvider gzdeflate DEFLATE Content-Type $text/
+ {{{
+ FilterProvider ucase CaseFilter Content-Type $text/
  
-   ProxyPass /proxy/ucase/ http://127.0.0.1/deflated/
+ ProxyPass /proxy/ucase/ http://127.0.0.1/deflated/
-   <Location "/proxy/ucase/">
+ <Location "/proxy/ucase/">
-       FilterChain +gzinflate +ucase +gzdeflate
+     FilterChain +gzinflate +ucase
-   </Location>
+ </Location>
+ }}}
  
-   127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
514 AE:"gzip,deflate" TE:"-" CE:"-"
-   127.0.0.1 "GET /proxy/ucase/LICENSE.txt HTTP/1.1" 200 9600 (9934) TE:"-" CE:"gzip" <--
428 AE:"gzip,deflate" TE:"-" CE:"-"
+ And the resulting log shows that the response to the client remains undeflated, and
+ if you inspect it in the browser, it's irritatingly in all caps;
  
+ {{{
+ 127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
513 AE:"gzip,deflate" TE:"-" CE:"-"
+ 127.0.0.1 "GET /proxy/ucase/LICENSE.txt HTTP/1.1" 200 39858 (40219) TE:"chunked" CE:"-"
<-- 427 AE:"gzip,deflate" TE:"-" CE:"-"
+ }}}
  
-   LogFormat "%h \"%r\" %>s %b (%O) TE:\"%{Transfer-Encoding}o\" CE:\"%{Content-Encoding}o\"
<-- %I AE:\"%{Accept-Encoding}i\" TE:\"%{Transfer-Encoding}i\" CE:\"%{Content-Encoding}i\""
compression
-   FilterProvider gzinflate INFLATE resp=Content-Encoding $gzip
+ Building on those two examples, we'll take this one step further, and ensure the
+ final response can in fact be deflated to the client, so we'll name our redeflate
+ filter gzdeflate and apply all three filters, in specific order, to inflate,
+ manipulate, and re-deflate the response;
  
+ {{{
+ FilterProvider gzdeflate DEFLATE Content-Type $text/
-   <VirtualHost 192.168.0.254:80>
-       ErrorLog logs/proxyerror.log
-       CustomLog logs/proxyaccess.log common
-       CustomLog logs/proxycompression.log compression
-       ProxyRequests On
-       ProxyVia On
  
+ ProxyPass /proxy/ucase/ http://127.0.0.1/deflated/
+ <Location "/proxy/ucase/">
+     FilterChain +gzinflate +ucase +gzdeflate
+ </Location>
+ }}}
-       <Proxy http:*>
-           FilterChain gzinflate
-           Order deny,allow
-           Deny from all
-           Allow from 127. 10. 192.168.
-       </Proxy>
-   </VirtualHost>
  
+ Note a fun side effect on deflation of upper-casing the document;
+ 
+ {{{
+ 127.0.0.1 "GET /deflated/LICENSE.txt HTTP/1.1" 200 10629 (10964) TE:"-" CE:"gzip" <--
514 AE:"gzip,deflate" TE:"-" CE:"-"
+ 127.0.0.1 "GET /proxy/ucase/LICENSE.txt HTTP/1.1" 200 9600 (9934) TE:"-" CE:"gzip" <--
428 AE:"gzip,deflate" TE:"-" CE:"-"
+ }}}
+ 
+ With only 26 alpha characters rather than 52 (upper and lower case) it compresses down
+ more efficiently than the back end had provided it to us.
+ 
+ We can actually do this for forward proxies as well.  Here is an example proxy server 
+ which inflating every C-E:gzip compressed response and serves only plain text to the 
+ clients using our proxy, we save about 10%.
+ 
+ {{{
+ LogFormat "%h \"%r\" %>s %b (%O) TE:\"%{Transfer-Encoding}o\" CE:\"%{Content-Encoding}o\"
<-- %I AE:\"%{Accept-Encoding}i\" TE:\"%{Transfer-Encoding}i\" CE:\"%{Content-Encoding}i\""
compression
+ FilterProvider gzinflate INFLATE resp=Content-Encoding $gzip
+ 
+ <VirtualHost 192.168.0.254:80>
+     ErrorLog logs/proxyerror.log
+     CustomLog logs/proxyaccess.log common
+     CustomLog logs/proxycompression.log compression
+     ProxyRequests On
+     ProxyVia On
+ 
+     <Proxy http:*>
+         FilterChain gzinflate
+         Order deny,allow
+         Deny from all
+         Allow from 127. 10. 192.168.
+     </Proxy>
+ </VirtualHost>
+ }}}
+ 

Mime
View raw message