trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Rushford <jrushf...@apache.org>
Subject Re: Downtime when traffic_ctl config reload
Date Tue, 15 Jan 2019 15:24:59 GMT
You could also enable parent.config debug logging that would help in
troubleshooting this

in records.config set:

CONFIG proxy.config.diags.debug.enabled INT 1
CONFIG proxy.config.diags.debug.tags STRING parent_select

# traffic_ctl config reload

On 1/12/19, 10:15 AM, "Miles Libbey" <mlibbey@apache.org> wrote:



    A loop would indicate its not matching parent.config then. I've never

    used an IP in a map. In


https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html

    it has a

    dest_ip

    field that would seem to make more sense than dest_domain|host for

    that map rule.



    You'll also need a map http://52.91.93.206/ http://someorigiin on your
parent...

    miles



    On Fri, Jan 11, 2019 at 10:49 PM Hobin Yoon <hobinyoon@gmail.com> wrote:

    >

    > Now, I'm getting

    >

    > Cycle Prohibited

    > ________________________________

    > Description: Could not process your request for the document because
it would cause an HTTP proxy cycle. Please check the URL and your browser's
proxy settings.

    >

    > I think it's because remap.config has the same ip addresses ... I
tried to comment out the map rule, but got the same error...

    >

    > On child node: (It's ip address is 52.91.93.206)

    >   records.config

    >     CONFIG proxy.config.http.parent_proxy_routing_enable INT 1

    >     CONFIG proxy.config.http.cache.http INT 0

    >   remap.config:

    >     map http://52.91.93.206 http://52.91.93.206

    >   parent.config:

    >    dest_domain=52.91.93.206 scheme=http parent="18.212.133.3:80,
34.207.101.159:80" round_robin=consistent_hash go_direct=false

    >

    > On parent nodes:

    >   records.config

    >     CONFIG proxy.config.http.parent_proxy_routing_enable INT 0

    >     CONFIG proxy.config.http.cache.http INT 1

    >

    > Hobin

    >

    > On Fri, Jan 11, 2019 at 5:31 PM Miles Libbey <mlibbey@apache.org>
wrote:

    >>

    >> First thing I'd check is
proxy.config.http.parent_proxy_routing_enable

    >> on the children :)

    >>

    >> On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <hobinyoon@gmail.com>
wrote:

    >> >

    >> > It is disabled. I just tried with both on and off, but none of
them worked. What bothers me is ATS doesn't seem to be reading
parent.config at all since I don't get any error when I intentionally made
a typo in parent.config.

    >> >

    >> > Hobin

    >> >

    >> > On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zwoop@apache.org>
wrote:

    >> >>

    >> >>

    >> >>

    >> >> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <hobinyoon@gmail.com>
wrote:

    >> >>

    >> >> Ok, I added the rule in remap.config. Now, the requests are going
to "origin". But, they bypasses parent nodes; I don't see any requests on
either of the parent nodes.

    >> >>

    >> >> Here is my system setup for test. "child" is a pure load balancer
and doesn't cache any data items. parent1 and parent2 do cache.

    >> >> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin
(Apache2)

    >> >>                                    └- parent2 (ATS) -┘

    >> >>

    >> >> Here is the configuration in "child" node.

    >> >>

    >> >> remap.config

    >> >> map / http://origin

    >> >>

    >> >> parent.config

    >> >> dest_domain=origin scheme=http parent="parent1:80;parent2:80"
round_robin=consistent_hash

    >> >>

    >> >>

    >> >>

    >> >> Do you have “pristine” host headers enabled? The mapping in
parent selection happens on the “remapped” URL, but I’m pretty sure it’d be
different if you have pristine host headers disabled.

    >> >>

    >> >> That much said, I’d run with a diagnostics (-T) on “parent”.

    >> >>

    >> >> — Leif

    >> >>

    >> >>

    >> >>

    >> >> records.config

    >> >> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1

    >> >>

    >> >> Hobin

    >> >>

    >> >> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zwoop@apache.org>
wrote:

    >> >>>

    >> >>>

    >> >>>

    >> >>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <hobinyoon@gmail.com>
wrote:

    >> >>>

    >> >>> Should you still specify the mapping in remap.config when you
have the mapping in parent.config?

    >> >>>

    >> >>>

    >> >>> Yes. Unless you run an open proxy, and then you have to turn off
“require remap”. Not recommended for almost all cases.

    >> >>>

    >> >>> — leif

    >> >>>

    >> >>>

    >> >>>

    >> >>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <
randallmeyer@yahoo.com> wrote:

    >> >>>>

    >> >>>>

    >> >>>> Looking at your remap.config in that repo, I see that all the
mappings are all commented out. Was that intentional?

    >> >>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
hobinyoon@gmail.com> wrote:

    >> >>>>

    >> >>>>

    >> >>>> Yes, they are also ATS nodes and they worked fine when requests
were made to them directly.

    >> >>>>

    >> >>>> I uploaded the config files here, if you could take a look:
https://gitlab.com/hobinyoon/trafficserver-config

    >> >>>>

    >> >>>> Hobin

    >> >>>>

    >> >>>>

    >> >>>>

    >> >>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <mlibbey@apache.org>
wrote:

    >> >>>>

    >> >>>> Typically that error means the request doesn't match a rule
in

    >> >>>> remap.config. Is the error coming from the first hop or one
of
the

    >> >>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS
nodes,

    >> >>>> are they configured to accept the requests they are getting?

    >> >>>>

    >> >>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <hobinyoon@gmail.com>
wrote:

    >> >>>> >

    >> >>>> > With parent.config

    >> >>>> >   dest_domain=. scheme=http parent="123.123.123.1:80,
123.123.123.2:80" round_robin=consistent_hash go_direct=false

    >> >>>> >

    >> >>>> > I'm getting

    >> >>>> >

    >> >>>> > Not Found on Accelerator

    >> >>>> > ________________________________

    >> >>>> > Description: Your request on the specified host was not
found. Check the location and try again.

    >> >>>> >

    >> >>>> > I must be missing something ...

    >> >>>> >

    >> >>>> > Hobin

    >> >>>> >

    >> >>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <
hobinyoon@gmail.com> wrote:

    >> >>>> >>

    >> >>>> >> Alan, that is the only map rule we have with the varying
number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1
200 OK" for the requests. I didn't check what it returned.

    >> >>>> >>

    >> >>>> >> Miles, I'll check out parent plugin!

    >> >>>> >>

    >> >>>> >> Hobin

    >> >>>> >>

    >> >>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
solidwallofcode@oath.com> wrote:

    >> >>>> >>>

    >> >>>> >>> It could be an artifact of reloading plugin configurations
if you have a lot of remap rules with plugins, although internally ATS
should do the load and then swap the configuration. During the down time,
does ATS process any traffic, or there is traffic but no caching?

    >> >>>> >>>

    >> >>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <
mlibbey@apache.org> wrote:

    >> >>>> >>>>

    >> >>>> >>>> We don't experience downtime when using traffic_ctl
config
reload (we

    >> >>>> >>>> use that ~daily).

    >> >>>> >>>>

    >> >>>> >>>> We don't use the balancer plugin. Instead,
we use
parent.config

    >> >>>> >>>> (
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
)

    >> >>>> >>>> to achieve the same consistent hash. Your config
would
translate to

    >> >>>> >>>>

    >> >>>> >>>> remap.config

    >> >>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so

    >> >>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true

    >> >>>> >>>>

    >> >>>> >>>> parent.config

    >> >>>> >>>> dest_domain=. scheme=http

    >> >>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
123.123.123.4:80"

    >> >>>> >>>> round_robin=consistent_hash go_direct=false

    >> >>>> >>>> dest_domain=. scheme=https

    >> >>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,
123.123.123.3:443,123.123.123.4:443"

    >> >>>> >>>> round_robin=consistent_hash go_direct=false

    >> >>>> >>>>

    >> >>>> >>>> miles

    >> >>>> >>>>

    >> >>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon
<
hobinyoon@gmail.com> wrote:

    >> >>>> >>>> >

    >> >>>> >>>> > Hi,

    >> >>>> >>>> >

    >> >>>> >>>> > We are noticing there is quite a bit of
delay when we
reload the config with traffic_ctl config reload. The delay is up to about
30 seconds, during which period we don't get any caching. We are using
consistency hashing plugin. The number of nodes changes dynamically between
5 to 30.

    >> >>>> >>>> >

    >> >>>> >>>> > Here is an example balancer (consistent
hash)
configuration in remap.config.

    >> >>>> >>>> >

    >> >>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
@pparam=--include-params=p0,p1 @pparam=--sort-params=true
@plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
@pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...

    >> >>>> >>>> >

    >> >>>> >>>> > Is this downtime normal? How do you guys
avoid the
service downtime while reconfiguring the cache nodes in the cluster?

    >> >>>> >>>> >

    >> >>>> >>>> > Hobin

    >> >>>> >>>> >

    >> >>>> >>>> >

    >> >>>> >>>> >

    >> >>>> >>>> >

    >> >>>> >>>> >

    >> >>>> >>>

    >> >>>> >>>

    >> >>>> >>>

    >> >>>> >>> --

    >> >>>> >>> Beware the fisherman who's casting out his line
in to a
dried up riverbed.

    >> >>>> >>> Oh don't try to tell him 'cause he won't believe.
Throw
some bread to the ducks instead.

    >> >>>> >>> It's easier that way. - Genesis : Duke : VI 25-28

    >> >>>

    >> >>>

    >> >>

Mime
View raw message