trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shu Kit Chan <chanshu...@gmail.com>
Subject Re: ESI pages return Content-Length: 0 and empty entity body
Date Sat, 18 Aug 2012 06:13:36 GMT
Yes. The plugin uses TSFetchUrl to fetch for each of the ESI Includes,
which only works if the src of the ESI include is accounted for in
remap.config

Thanks.

Kit

On Fri, Aug 17, 2012 at 10:11 PM, James Peach <jpeach@apache.org> wrote:
> Bringing this thread back to the mailing list so the useful info below can be archived
:)
>
> On 17/08/2012, at 6:36 PM, Shu Kit Chan <chanshukit@gmail.com> wrote:
>
>> Andrew,
>>
>> Sorry. I accidentally press the "Reply" instead of "Reply All" last time
>>
>> I took a longer look at your original email and I tried to simulate in
>> my own setup. I think there is at least two thing missing definitely.
>>
>> #1) In remap.config, you need to have
>>
>> map / http://dev1.test/~ho/proxied
>> map http://dev1.test/~ho/proxied http://dev1.test/~ho/proxied
>>
>> It's a bit counter-intuitive but that's just how it works for making
>> internal call with ATS (i.e. the use of TSFetchUrl).
>
> Yeh that seems a bit redundant to me too.
>
>>
>> #2) Unfortunately, not everything in esi spec works fine in this
>> plugin. So a bit of documentation is needed. Specifically, "onerror"
>> is not working fine. So with the problem in #1, the YTS do not know
>> where to get your snippet and don't care about "onerror". Thus it just
>> error out with an empty response for you (not a good way to end, i
>> might add).
>>
>> If you try the following, it should work
>>
>> <html>
>> <head>
>>  <title>ESI test</title>
>> </head>
>> <body>
>>  <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p>
>>  <esi:try><esi:attempt><esi:include
>> src="http://dev1.test/~ho/snippet.cgi"/></esi:attempt><esi:except>Sorry!</esi:except></esi:try>
>> </body>
>> </html>
>>
>> Anyway, let me know how it works for u. I am still getting my upstream
>> merged fork to compile.
>> If you check the email archieve, you will see I help out a guy to get
>> ESI working for him a couple months ago. Not too sure if he finally
>> uses ESI in production or not. Yahoo! is definitely using ESI in their
>> production. The code that ATS has is originally donated from yahoo!
>> But since then Y! did make a few fixes and the Y! people are also
>> aware of the fact that some parts of the spec are not working. And
>> they have their own documentation of what part of the spec to use
>> versus what part you should not use. Hope this info helps.
>
> I just hooked up the ESI unit tests. There's a bunch that fail and I'd be happy to merge
patches to fix any of these. Hopefull this will make it clearer what works and what doesn't.
>
>>
>> Thanks.
>>
>> Kit
>>
>> On Fri, Aug 17, 2012 at 5:36 PM, James Peach <jamespeach@me.com> wrote:
>>> Add Andrew to CC ...
>>>
>>> On Aug 17, 2012, at 5:34 PM, Shu Kit Chan <chanshukit@gmail.com> wrote:
>>>
>>>> i am spending some times to fetch from upstream and trying things out.
>>>> Will report back later.
>>>>
>>>> Meanwhile...
>>>>
>>>> Can you show the squid.log of ats during your run?
>>>> Also, I think if you set
>>>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi*
>>>>
>>>> There will be more meaningful debug messages in traffic.out for us as well.
>>>>
>>>> Thanks.
>>>>
>>>> Kit
>>>>
>>>> On Fri, Aug 17, 2012 at 5:10 PM, James Peach <jamespeach@me.com> wrote:
>>>>> On Aug 17, 2012, at 4:50 PM, Andrew Ho <ho@groupon.com> wrote:
>>>>>
>>>>>> Hi James,
>>>>>>
>>>>>> Thanks for the offer for help! In this case, though, I think it's
Kit who needs to update his fork to merge in the upstream changes (specifically, your commit
053cc43e which added the build bits for plugin/experimental/esi). And that's just so his fork
will build from a fresh checkout—the problem that I'm seeing hasn't been diagnosed yet.
>>>>>
>>>>> There's a bit of both ... I've merged some of Kit's changes but not all.
It's possible that the result is not working. I'm disappointed to hear that Kit's fork isn't
working for you. I'll see whether I can dig into it a bit tonight ...
>>>>>
>>>>>>
>>>>>> As for the deprecated warnings, I'll be happy to dig into that and
propose a patch after I get basic ESI working.  :)
>>>>>>
>>>>>> Humbly,
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>> --
>>>>>> Andrew Ho
>>>>>> Senior Software Developer
>>>>>> ho@groupon.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 17, 2012 at 4:18 PM, James Peach <jamespeach@me.com>
wrote:
>>>>>>
>>>>>> On Aug 17, 2012, at 4:14 PM, Andrew Ho <ho@groupon.com> wrote:
>>>>>>
>>>>>>> Hi Kit,
>>>>>>>
>>>>>>> Heh, I just stole the one that was checked in to the upstream
repository earlier this week:
>>>>>>>
>>>>>>> wget -Oplugins/experimental/esi/Makefile.am https://raw.github.com/apache/trafficserver/master/plugins/experimental/esi/Makefile.am
>>>>>>> perl -pi.orig -e 's,experimental/lua,experimental/esi,' configure.ac
>>>>>>> autoreconf -i
>>>>>>> CFLAGS='-fPIC -Wno-deprecated' CXXFLAGS='-fPIC -Wno-deprecated'
./configure
>>>>>>> (cd plugins/experimental/esi && make)
>>>>>>>
>>>>>>> (The silly bit with Perl is because the default configure.ac
in your fork had also not yet been updated.) Note that I had to add -Wno-deprecated because
otherwise the -Wall resulted in some deprecation warnings.
>>>>>>>
>>>>>>> I'll try this out and also give your Makefile a try (but you
might want to merge the upstream changes to the build system, it's nice that there's a Makefile.am
for esi now).
>>>>>>
>>>>>> Hey, if you guys can pop up on IRC I should be able to turn the merges
around pretty quickly and we can verify what is working ... I'll be on later tonight after
8pm Pacific
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Humbly,
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Ho
>>>>>>> Senior Software Developer
>>>>>>> ho@groupon.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 17, 2012 at 3:49 PM, Shu Kit Chan <chanshukit@gmail.com>
wrote:
>>>>>>> Just added the Makefile I use.
>>>>>>> You can try it out. It is nothing fancy. A pretty simple one.
>>>>>>>
>>>>>>> On Fri, Aug 17, 2012 at 3:09 PM, Andrew Ho <ho@groupon.com>
wrote:
>>>>>>>> Hi Kit,
>>>>>>>>
>>>>>>>> Thanks for your quick response. I had come across your blog
entries and
>>>>>>>> previous mailing list posts and they were quite useful for
me so far.
>>>>>>>>
>>>>>>>> I noticed that there is neither a Makefile nor a Makefile.am
in your
>>>>>>>> repository. How do you usually build the plugin from source?
>>>>>>>>
>>>>>>>>
>>>>>>>> Humbly,
>>>>>>>>
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andrew Ho
>>>>>>>> Senior Software Developer
>>>>>>>> ho@groupon.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 17, 2012 at 1:31 PM, Shu Kit Chan <chanshukit@gmail.com>
wrote:
>>>>>>>>>
>>>>>>>>> The current latest on git is still "experimental".
>>>>>>>>> I have made a bunch of fixes which James Peach is helping
me to verify.
>>>>>>>>> The conversation can be found here -
>>>>>>>>> https://issues.apache.org/jira/browse/TS-1249
>>>>>>>>>
>>>>>>>>> If you really want to try, you can try to compile the
forked version I
>>>>>>>>> have.
>>>>>>>>>
>>>>>>>>> https://github.com/shukitchan/trafficserver/tree/master/plugins/experimental/esi
>>>>>>>>> and i hope it should work for you.
>>>>>>>>>
>>>>>>>>> It is at least working quite well for me.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Kit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 17, 2012 at 12:42 PM, Andrew Ho <ho@groupon.com>
wrote:
>>>>>>>>>> Hi Traffic Server users,
>>>>>>>>>>
>>>>>>>>>> I can't seem to get ESIs working on Traffic Server.
When I request an
>>>>>>>>>> ESI
>>>>>>>>>> enabled page, I get a 200 OK status, but "Content-Length:
0" and an
>>>>>>>>>> empty
>>>>>>>>>> entity body. There is no evidence on the origin server
side that the ESI
>>>>>>>>>> sub-pages were ever loaded. Does this symptom sound
familiar to anybody?
>>>>>>>>>>
>>>>>>>>>> The gory details follow...
>>>>>>>>>>
>>>>>>>>>> I am using Traffic Server version 3.2.0 (Linux distribution
is CentOS
>>>>>>>>>> 5.4,
>>>>>>>>>> and the test hosts are in Amazon EC2):
>>>>>>>>>>
>>>>>>>>>> % traffic_server --version
>>>>>>>>>> [TrafficServer] using root directory '/usr/local'
>>>>>>>>>> Apache Traffic Server - traffic_server - 3.2.0 -
(build # 7719 on Aug  7
>>>>>>>>>> 2012 at 19:06:21)
>>>>>>>>>>
>>>>>>>>>> The ESI plugin I built was from commit 025cd279 of
repository version:
>>>>>>>>>> https://git-wip-us.apache.org/repos/asf/trafficserver.git/
>>>>>>>>>>
>>>>>>>>>> The following are the configuration changes I enacted
to the defaults.
>>>>>>>>>> In
>>>>>>>>>> plugin.config, I added this line to enable the ESI
plugin:
>>>>>>>>>>
>>>>>>>>>> esi.so
>>>>>>>>>>
>>>>>>>>>> In records.config, I added these lines to enable
debug logging:
>>>>>>>>>>
>>>>>>>>>> CONFIG proxy.config.diags.debug.enabled INT 1
>>>>>>>>>> CONFIG proxy.config.diags.debug.tags STRING plugin_esi_intercept*
>>>>>>>>>>
>>>>>>>>>> In remap.config, I added this line to add an origin
server:
>>>>>>>>>>
>>>>>>>>>> map / http://dev1.test/~ho/proxied
>>>>>>>>>>
>>>>>>>>>> dev1.test is an internal development host which runs
a vanilla Apache
>>>>>>>>>> HTTPD
>>>>>>>>>> configured to serve up content from ~/public_html
directories. In
>>>>>>>>>> dev1.test:~ho/public_html/proxied, I have a couple
CGI scripts
>>>>>>>>>> (snippet.cgi
>>>>>>>>>> emits a bare HTML fragment, esi.cgi emits an "X-Esi:
1" header and an
>>>>>>>>>> HTML
>>>>>>>>>> document that includes snippet.cgi with a fully-qualified
URL):
>>>>>>>>>>
>>>>>>>>>> % cat ~ho/public_html/proxied/snippet.cgi
>>>>>>>>>> #!/usr/local/bin/perl
>>>>>>>>>>
>>>>>>>>>> my $body = <<'EndHTML';
>>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p>
>>>>>>>>>> EndHTML
>>>>>>>>>>
>>>>>>>>>> print "Content-Type: text/html\n",
>>>>>>>>>>     "Content-Length: ", length($body), "\n",
>>>>>>>>>>     "\n",
>>>>>>>>>>     $body;
>>>>>>>>>>
>>>>>>>>>> % cat ~ho/public_html/proxied/esi.cgi
>>>>>>>>>> #!/usr/local/bin/perl
>>>>>>>>>>
>>>>>>>>>> my $body = <<'EndHTML';
>>>>>>>>>> <html>
>>>>>>>>>> <head>
>>>>>>>>>> <title>ESI test</title>
>>>>>>>>>> </head>
>>>>>>>>>> <body>
>>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p>
>>>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"
>>>>>>>>>> onerror="continue"/>
>>>>>>>>>> </body>
>>>>>>>>>> </html>
>>>>>>>>>> EndHTML
>>>>>>>>>>
>>>>>>>>>> print "Content-Type: text/html\n",
>>>>>>>>>>     "Content-Length: ", length($body), "\n",
>>>>>>>>>>     "X-Esi: 1\n",
>>>>>>>>>>     "\n",
>>>>>>>>>>     $body;
>>>>>>>>>>
>>>>>>>>>> From the host running Traffic server, I can definitely
load those URLs:
>>>>>>>>>>
>>>>>>>>>> % curl -si http://dev1/~ho/proxied/snippet.cgi
>>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:20 GMT
>>>>>>>>>> Server: Apache/2.2.17
>>>>>>>>>> Content-Length: 71
>>>>>>>>>> Content-Type: text/html
>>>>>>>>>>
>>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p>
>>>>>>>>>>
>>>>>>>>>> % curl -si http://dev1/~ho/proxied/esi.cgi
>>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:22 GMT
>>>>>>>>>> Server: Apache/2.2.17
>>>>>>>>>> X-Esi: 1
>>>>>>>>>> Content-Length: 219
>>>>>>>>>> Content-Type: text/html
>>>>>>>>>>
>>>>>>>>>> <html>
>>>>>>>>>> <head>
>>>>>>>>>> <title>ESI test</title>
>>>>>>>>>> </head>
>>>>>>>>>> <body>
>>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/esi.cgi</tt>.</p>
>>>>>>>>>> <esi:include src="http://dev1.test/~ho/snippet.cgi"
>>>>>>>>>> onerror="continue"/>
>>>>>>>>>> </body>
>>>>>>>>>> </html>
>>>>>>>>>>
>>>>>>>>>> And when I issue requests on the hosts running Traffic
Server, reverse
>>>>>>>>>> proxying is generally working, as I can load snippet.cgi:
>>>>>>>>>>
>>>>>>>>>> % curl -si http://localhost:8080/snippet.cgi
>>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>>> Date: Fri, 17 Aug 2012 19:24:55 GMT
>>>>>>>>>> Server: ATS/3.2.0
>>>>>>>>>> Content-Length: 71
>>>>>>>>>> Content-Type: text/html
>>>>>>>>>> Age: 0
>>>>>>>>>> Connection: keep-alive
>>>>>>>>>>
>>>>>>>>>> <p>This is <tt>dev1.test:~ho/public_html/proxied/snippet.cgi</tt>.</p>
>>>>>>>>>>
>>>>>>>>>> However, I get a "Content-Length: 0" and an empty
entity body on the ESI
>>>>>>>>>> enabled request to esi.cgi (despite a 200 OK status!):
>>>>>>>>>>
>>>>>>>>>> % curl -si http://localhost:8080/esi.cgi
>>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>>> Date: Fri, 17 Aug 2012 19:25:15 GMT
>>>>>>>>>> Server: ATS/3.2.0
>>>>>>>>>> Content-Type: text/html
>>>>>>>>>> Content-Length: 0
>>>>>>>>>> Connection: keep-alive
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Here is the logging information captured in traffic.out
for that
>>>>>>>>>> request:
>>>>>>>>>>
>>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [setupServerIntercept] Setup server intercept successfully
>>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Received net accept event
>>>>>>>>>> [Aug 17 15:30:14.029] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [init] ContData initialized!
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Received read ready event
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [handleRead] Parsed header
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [handleRead] Got content length as 257
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Received read ready event
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [handleRead] Appending 356 bytes to body
>>>>>>>>>> [Aug 17 15:30:14.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [handleRead] Appending 257 bytes to body
>>>>>>>>>> [Aug 17 15:30:44.030] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Received read complete/eos event
104
>>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [processRequest] Wrote reply of size 761
>>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Processed request successfully
>>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [serverIntercept] Completed request processing. Shutting
down...
>>>>>>>>>> [Aug 17 15:30:44.031] Server {0x2b895be76040} DIAG:
>>>>>>>>>> (plugin_esi_intercept)
>>>>>>>>>> [~ContData] Destroying continuation data
>>>>>>>>>>
>>>>>>>>>> On the server side, I only see the single request
for esi.cgi (and not
>>>>>>>>>> for
>>>>>>>>>> snippet.cgi):
>>>>>>>>>>
>>>>>>>>>> 6071 dev1 x.x.x.x - - [17/Aug/2012:15:30:13 -0400]
"GET
>>>>>>>>>> /~ho/proxied/esi.cgi
>>>>>>>>>> HTTP/1.1" 200 219 "-" "-" 193 354 5083
>>>>>>>>>>
>>>>>>>>>> We are evaluating Traffic Server specifically for
ESI support, so this
>>>>>>>>>> bug
>>>>>>>>>> is quite a blocker for me. I've tried a few different
things based on
>>>>>>>>>> searching the Traffic Server mailing list in the
past (for example, I
>>>>>>>>>> made
>>>>>>>>>> sure deflate was disabled on the origin server),
and now I'm stuck. Does
>>>>>>>>>> anybody know what is going on, or can someone suggest
a next idea for me
>>>>>>>>>> to
>>>>>>>>>> debug the problem? Thanks!
>>>>>>>>>>
>>>>>>>>>> Humbly,
>>>>>>>>>>
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Andrew Ho
>>>>>>>>>> Senior Software Developer
>>>>>>>>>> ho@groupon.com
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>

Mime
View raw message