cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Fontana Oscarsson (Jira)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-16718) Changing listen_address with prefer_local may lead to issues
Date Wed, 01 Sep 2021 07:10:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-16718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407896#comment-17407896
] 

Simon Fontana Oscarsson commented on CASSANDRA-16718:
-----------------------------------------------------

Hi [~brandon.williams],

Jan asked me to answer since I have a deeper knowledge in the issue at hand.

We deploy Cassandra in a Kubernetes cluster. To achieve external connectivity to remote DCs
(Cassandra DCs in different Kubernetes clusters) we use MetalLB (one LoadBalancer Service
per Pod). Some of our deployments use traffic policy "Cluster" which doesn't preserve source
IP.
{quote}However, kube-proxy will obscure the source IP address of the connection when it does
load balancing, so your pod logs will show that external traffic appears to be coming from
the service’s leader node.
{quote}
- [https://metallb.universe.tf/usage/#cluster-traffic-policy]

We have a Cassandra plugin which uses the IP for its execution logic. With prefer_local=false
all traffic (incl. local traffic) will go through the load balancer and therefore obscures
the source IP. With prefer_local the source IP for local nodes is preserved so it's available
to our plugin.

Let me know if you need more details.

> Changing listen_address with prefer_local may lead to issues
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-16718
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16718
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Config
>            Reporter: Jan Karlsson
>            Priority: Normal
>             Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> Many container based solution function by assigning new listen_addresses when nodes are
stopped. Changing the listen_address is usually as simple as turning off the node and changing
the yaml file. 
> However, if prefer_local is enabled, I observed that nodes were unable to join the cluster
and fail with 'Unable to gossip with any seeds'. 
> Trace shows that the changing node will try to communicate with the existing node but
the response is never received. I assume it is because the existing node attempts to communicate
with the local address during the shadow round.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message