trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miles Libbey <mlib...@apache.org>
Subject Re: Query strings are not forwarded
Date Wed, 02 Nov 2016 16:03:17 GMT
Hi Randeep-
I think I'm confused about what you are trying to do now.

- I believe the cachekey plugin's ua-whitelist would change the cachekey
for the UserAgent's specified. So, in your example, I think if you run
curl -D- -o/dev/null  "http://jitp1.domain.com/dashw/zoom/manifest.mpd" -H
'User-Agent: stagefright/1'  -H 'X-Debug: X-Cache-Key' | grep X-Cache-Key

you'd get
X-Cache-Key: /jitp1.domain.com/80/lukupplayer/dashw/zoom/manifest.mpd

- the cachekey's remove-all-params=true will remove the query string from
the cachekey (but, not from the request to the origin. So, if you run
curl -D- -o/dev/null  "http://jitp1.domain.com/dashw/zoom/manifest.mpd?foo=1"
-H 'User-Agent: stagefright/1'  -H 'X-Debug: X-Cache-Key' | grep X-Cache-Key
you'd get
X-Cache-Key: /jitp1.domain.com/80/lukupplayer/dashw/zoom/manifest.mpd
or instead you run
curl -D- -o/dev/null  "http://jitp1.domain.com/dashw/zoom/manifest.mpd?foo=1"
-H 'User-Agent: AppleWebKit/1'  -H 'X-Debug: X-Cache-Key' | grep X-Cache-Key
you'd get
X-Cache-Key: /jitp1.domain.com/80/dashw/zoom/manifest.mpd

I think you can get some cachekey debug statements if you set the following
variables in records.config (in the Traffic Server config directory):

CONFIG proxy.config.diags.debug.enabled INT 1
CONFIG proxy.config.diags.debug.tags STRING cachekey

miles


On Wed, Nov 2, 2016 at 5:59 AM, Randeep <randeep123@gmail.com> wrote:

> Hi,
>
> Its true. I started using this plugin for removing the query strings from
> the requests. But when I checked the documentation, I found that the same
> plugin can be used to whitelist user-agents also.
>
> https://docs.trafficserver.apache.org/en/6.2.x/admin-
> guide/plugins/cachekey.en.html?highlight=cachekey
>
> I have whitelisted only Stagefright. But for AppleWebkit also requests are
> served.
>
> cat remap.config
> map http://jitp1.domain.com/ http://bglrcache1.domain.com/
> @plugin=cachekey.so  @pparam=--ua-whitelist=lukupplayer:/opt/ats/etc/trafficserver/randeep.config
> @pparam=--remove-all-params=true
>
> cat /opt/ats/etc/trafficserver/randeep.config
> ^stagefright/1.*
>
> @Miles Libbey
> I have enabled Xdebug plugin.
>
> [root@cachetest trafficserver]# cat plugin.config
> background_fetch.so --config /opt/ats/etc/trafficserver/
> background_fetch.config
> *xdebug.so*
> [root@cachetest trafficserver]#
>
> [Nov  2 18:12:58.353] {0x2aafed658a40} STATUS: opened /opt/ats/var/log/
> trafficserver/diags.log
> [Nov  2 18:12:58.376] Server {0x2aafed658a40} NOTE: <Plugin.cc:84
> (plugin_load)> loading plugin '/opt/ats/libexec/trafficserver/background_
> fetch.so'
> *[Nov  2 18:12:58.377] Server {0x2aafed658a40} NOTE: <Plugin.cc:84
> (plugin_load)> loading plugin '/opt/ats/libexec/trafficserver/xdebug.so'*
>
> curl -v  "http://jitp1.domain.com/dashw/zoom/manifest.mpd" -H
> 'User-Agent: AppleWebKit'  -H 'X-Debug: X-Cache-Key'
> * About to connect() to jitp1.domain.com port 80 (#0)
> *   Trying 172.16.0.51... connected
> * Connected to jitp1.domain.com (172.16.0.51) port 80 (#0)
> > GET /dashw/zoom/manifest.mpd HTTP/1.1
> > Host: jitp1.domain.com
> > Accept: */*
> > User-Agent: AppleWebKit
> > X-Debug: X-Cache-Key
> >
> < HTTP/1.1 200 OK
> < Content-Length: 2084
> < Content-Type: application/dash+xml
> < Access-Control-Allow-Origin: *
> < Cache-Control: max-age=5
> < Date: Wed, 02 Nov 2016 12:43:23 GMT
> < Age: 22
> < Server: ATS/6.2.0
> < Connection: keep-alive
> *< X-Cache-Key: /jitp1.domain.com/80/dashw/zoom/manifest.mpd
> <http://jitp1.domain.com/80/dashw/zoom/manifest.mpd>*
> <
> <?xml version="1.0" encoding="UTF-8"?><MPD
> *** Data Truncated***
>
> Can we get any specific log for cachekey whitelist check?
>
> Regards,
> Randeep
>
> On Fri, Oct 28, 2016 at 10:23 PM, Miles Libbey <mlibbey@apache.org> wrote:
>
>> On Fri, Oct 28, 2016 at 6:04 AM, Sudheer Vinukonda
>> <sudheervinukonda@yahoo.com.invalid> wrote:
>> > Not sure to follow - what exactly are you trying to do? Afaik, cachekey
>> plugin only manipulates the otherwise default cache key (which is the full
>> request URL) used against the configured cache object(s) and modifies it as
>> specified.
>> >
>> > I don't believe it can "block" any requests or user-agents.
>> >
>> >> On Oct 28, 2016, at 4:17 AM, Randeep <randeep123@gmail.com> wrote:
>> >>
>> >> Hi Sudheer,
>> >>
>> >> Sorry. I had added it. While pasting here somehow it was not selected.
>> I
>> >> was was checking the whitelisting of a user agent. Query string was
>> >> working.
>> >>
>> >> map http://jitp1.domain.com/ http://bglrcache1.domain.com/
>> >> @plugin=cachekey.so
>> >> @pparam=--ua-whitelist=player:/opt/ats/etc/trafficserver/ran
>> deep.config
>> >> @pparam=--ua-capture=(stagefright\/[^\s]*).*
>> >> @pparam=--remove-all-params=true
>> >>
>> >> Still its not working. It is not blocking the other user agents.
>> >>
>> >> [root@cachetest ~]# curl -v  "
>> >> http://jitp1.domain.com/dashw/zoom/manifest.mpd" -H 'User-Agent:
>> >> AppleWebKit'  -H 'X-Debug: X-Cache-Key'
>> >> * About to connect() to jitp1.domain.com port 80 (#0)
>> >> *   Trying 172.16.0.51... connected
>> >> * Connected to jitp1.domain.com (172.16.0.51) port 80 (#0)
>> >>> GET /dashw/zoom/manifest.mpd HTTP/1.1
>> >>> Host: jitp1.domain.com
>> >>> Accept: */*
>> >>> User-Agent: AppleWebKit
>> >>> X-Debug: X-Cache-Key
>>
>> It doesn't look like you have the X-Debug plugin enabled, so the
>> Cache-Key isn't being printed here, making it hard to figure out.
>> Could you enable it?
>>
>>
>> >>>
>> >> < HTTP/1.1 200 OK
>> >> < Content-Length: 2082
>> >> < Content-Type: application/dash+xml
>> >> < Access-Control-Allow-Origin: *
>> >> < Cache-Control: max-age=5
>> >> < Date: Fri, 28 Oct 2016 11:14:31 GMT
>> >> < Age: 19
>> >> < Server: ATS/6.2.0
>> >> < Connection: keep-alive
>> >> <
>> >> <?xml version="1.0" encoding="UTF-8"?><MPD xmlns:xsi="
>> >> http://www.w3.org/2001/XMLSchema-instance"
>> >> xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:cenc="urn:mpeg:cenc:2013"
>> >> ***Truncated***
>> >>
>> >> Regards,
>> >> Randeep
>> >>
>> >> On Fri, Oct 28, 2016 at 4:34 PM, Sudheer Vinukonda <
>> >> sudheervinukonda@yahoo.com> wrote:
>> >>
>> >>> Hmm...where's the pparam to strip query string?
>> >>>
>> >>> Please make sure to read through the documentation on cachekey.
>> >>> Specifically, the query param section you are looking for and the
>> examples.
>> >>>
>> >>> Here's a relevant extract -
>> >>>
>> >>> URI query parameters
>> >>> Ignore the query string (all query parameters)
>> >>>
>> >>> The following added to the remap rule will ignore the query, removing
>> it
>> >>> from the cache key.
>> >>>
>> >>> @plugin=cachekey.so @pparam=--remove-all-params=true
>> >>>
>> >>>
>> >>> On Oct 28, 2016, at 2:27 AM, Randeep <randeep123@gmail.com> wrote:
>> >>>
>> >>> Hi Miles Libbey,
>> >>>
>> >>> Thanks for pointing it out.  I changed it as below.
>> >>> map http://jitp1.domain.com/ http://bglrcache1.domain.com/
>> >>> @plugin=cachekey.so  @pparam=--ua-whitelist=player:
>> >>> /opt/ats/etc/trafficserver/randeep.config  @pparam=--ua-capture=(
>> >>> stagefright\/[^\s]*).*
>> >>>
>> >>> Still now luck.
>> >>>
>> >>> [root@cachetest ~]# curl -v  "http://jitp1.domain.com/
>> >>> dashw/zoom/manifest.mpd" -H 'User-Agent: stagefright/1.2
>> (Linux;Android
>> >>> 4.2.109)'  -H 'X-Debug: X-Cache-Key'
>> >>> * About to connect() to jitp1.domain.com port 80 (#0)
>> >>> *   Trying 172.16.0.51... connected
>> >>> * Connected to jitp1.domain.com (172.16.0.51) port 80 (#0)
>> >>>> GET /dashw/zoom/manifest.mpd HTTP/1.1
>> >>>> Host: jitp1.domain.com
>> >>>> Accept: */*
>> >>>> User-Agent: stagefright/1.2 (Linux;Android 4.2.109)
>> >>>> X-Debug: X-Cache-Key
>> >>>>
>> >>> < HTTP/1.1 200 OK
>> >>> < Content-Length: 2082
>> >>> < Content-Type: application/dash+xml
>> >>> < Access-Control-Allow-Origin: *
>> >>> < Cache-Control: max-age=5
>> >>> < Date: Fri, 28 Oct 2016 09:23:33 GMT
>> >>> < Age: 20
>> >>> < Server: ATS/6.2.0
>> >>> < Connection: keep-alive
>> >>> <
>> >>> <?xml version="1.0" encoding="UTF-8"?><MPD xmlns:xsi="
>> http://www.w3.org/
>> >>> 2001/XMLSchema-instance" xmlns="urn:mpeg:dash:schema:mpd:2011"
>> >>> ***Truncated****
>> >>>
>> >>> Regards,
>> >>> Randeep
>> >>>
>> >>>> On Fri, Oct 28, 2016 at 1:24 PM, Miles Libbey <mlibbey@apache.org>
>> wrote:
>> >>>>
>> >>>>> On Thu, Oct 27, 2016 at 11:41 PM, Randeep <randeep123@gmail.com>
>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I used the cachekey plugin as below. But its not working. Am
I
>> doing it
>> >>>>> wrong?
>> >>>>>
>> >>>>> My remap.config
>> >>>>> map http://jitp1.domain.com/ http://bglrcache1.domain.com/
>> >>>>> @pparam=--ua-whitelist=lukupplayer:/opt/ats/etc/trafficserve
>> r/randeep.config
>> >>>>> @pparam=--ua-capture=(stagefright\/[^\s]*).*
>> >>>>>
>> >>>>
>> >>>> shouldn't you have
>> >>>> @plugin=cachekey.so in there?
>> >>>> eg
>> >>>> map http://jitp1.domain.com/ http://bglrcache1.domain.com/
>> >>>> @plugin=cachekey.so @pparam=--ua-whitelist=...
>> >>>>
>> >>>>
>> >>>> cat /opt/ats/etc/trafficserver/randeep.config
>> >>>>> ^stagefright/1.*
>> >>>>>
>> >>>>> I am requesting the like this
>> >>>>> curl "http://jitp1.domain.com/dashw/abc/manifest.mpd" -H
>> 'User-Agent:
>> >>>>> stagefright/1.2 (Linux;Android 4.2.109)'  -H 'X-Debug: X-Cache-Key'
>> >>>>>
>> >>>>> But its not rejected!
>> >>>>> [root@cachetest ~]# tail -f /opt/ats/var/log/trafficserver
>> /squid.log
>> >>>>> |grep abc
>> >>>>> 1477636406.439 3 172.16.0.51 TCP_MISS/404 199 GET
>> >>>>> http://bglrcache1.domain.com/dashw/abc/manifest.mpd - PARENT_HIT/
>> >>>>> 172.16.0.40 text/plain
>> >>>>>
>> >>>>> Please guide.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Randeep
>> >>>>>
>> >>>>>> On Thu, Oct 27, 2016 at 5:02 PM, Randeep <randeep123@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> url http://jitp1.dmain.com/dashw/abpnews/manifest.mpd
>> >>>>>> <http://jitp1.lukup.com/dashw/abpnews/manifest.mpd?starttime
>> =1477454400000&stoptime=1477456200000> will
>> >>>>>> give the live content
>> >>>>>>
>> >>>>>> url http://jitp1.dmain.com/dashw/abpnews/manifest.mpd?startt
>> >>>>>> ime=1477454400000&stoptime=1477456200000
>> >>>>>> <http://jitp1.lukup.com/dashw/abpnews/manifest.mpd?starttime
>> =1477454400000&stoptime=1477456200000>
>> >>>>>> will give the recorded content from the starttime to endtime.
The
>> manifest
>> >>>>>> file will be for the specified time.
>> >>>>>>
>> >>>>>> but in that case if we cache the response without query
strings
>> for the
>> >>>>>> above, it might conflict with the live requests!
>> >>>>>>
>> >>>>>> I think its better not cache the manifest url with query
strings.
>> Only
>> >>>>>> the video chunks and manifest file without query string
we will
>> cache.
>> >>>>>> Is that possible?
>> >>>>>>
>> >>>>>> Regards,
>> >>>>>> Randeep
>> >>>>>>
>> >>>>>> On Thu, Oct 27, 2016 at 4:50 PM, Reindl Harald <
>> h.reindl@thelounge.net>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> Am 27.10.2016 um 13:05 schrieb Randeep:
>> >>>>>>>>
>> >>>>>>>> Our urls are like
>> >>>>>>>>
>> >>>>>>>> http://jitp1.dmain.com/dashw/abpnews/manifest.mpd?starttime=
>> >>>>>>>> 1477454400000&stoptime=1477456200000
>> >>>>>>>> <http://jitp1.lukup.com/dashw/abpnews/manifest.mpd?starttime
>> >>>>>>>> =1477454400000&stoptime=1477456200000>
>> >>>>>>>>
>> >>>>>>>> We need to cache only
>> >>>>>>>> http://jitp1.dmain.com/dashw/abpnews/manifest.mpd
>> >>>>>>>> <http://jitp1.lukup.com/dashw/abpnews/manifest.mpd?starttime
>> >>>>>>>> =1477454400000&stoptime=1477456200000>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> but *why* are your URLs than that way?
>> >>>>>>>
>> >>>>>>> if that params don't do anything remove them
>> >>>>>>>
>> >>>>>>> if you configrue your proxy to ignore params and always
deliver
>> the
>> >>>>>>> same content that will for sure break something else
and in doubt
>> it will
>> >>>>>>> break terrible
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Randeep
>> >>>>>> Mob: +919447831699[kerala]
>> >>>>>> Mob: +919880050349[B'lore]
>> >>>>>> http://twitter.com/Randeeppr
>> >>>>>> http://in.linkedin.com/in/randeeppr
>> >>>>>>
>> >>>>>>
>> >>>>>> [image: --]
>> >>>>>> Randeep Raman
>> >>>>>> [image: http://]about.me/Randeeppr
>> >>>>>> <http://about.me/Randeeppr>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Randeep
>> >>>>> Mob: +919447831699[kerala]
>> >>>>> Mob: +919880050349[B'lore]
>> >>>>> http://twitter.com/Randeeppr
>> >>>>> http://in.linkedin.com/in/randeeppr
>> >>>>>
>> >>>>>
>> >>>>> [image: --]
>> >>>>> Randeep Raman
>> >>>>> [image: http://]about.me/Randeeppr
>> >>>>> <http://about.me/Randeeppr>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Randeep
>> >>> Mob: +919447831699[kerala]
>> >>> Mob: +919880050349[B'lore]
>> >>> http://twitter.com/Randeeppr
>> >>> http://in.linkedin.com/in/randeeppr
>> >>>
>> >>>
>> >>> [image: --]
>> >>> Randeep Raman
>> >>> [image: http://]about.me/Randeeppr
>> >>> <http://about.me/Randeeppr>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Randeep
>> >> Mob: +919447831699[kerala]
>> >> Mob: +919880050349[B'lore]
>> >> http://twitter.com/Randeeppr
>> >> http://in.linkedin.com/in/randeeppr
>> >>
>> >>
>> >> [image: --]
>> >> Randeep Raman
>> >> [image: http://]about.me/Randeeppr
>> >> <http://about.me/Randeeppr>
>> >
>>
>
>
>
> --
> Randeep
> Mob: +919447831699[kerala]
> Mob: +919880050349[B'lore]
> http://twitter.com/Randeeppr
> http://in.linkedin.com/in/randeeppr
>
>
> [image: --]
> Randeep Raman
> [image: http://]about.me/Randeeppr
> <http://about.me/Randeeppr>
>
>

Mime
View raw message