spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jose Luis Pedrosa <Jose.Pedr...@microsoft.com.INVALID>
Subject Re: [External Sender] Re: Spark 2.4.1 on Kubernetes - DNS resolution of driver fails
Date Tue, 18 Jun 2019 15:38:24 GMT
Hi!

I am assuming you’re running it in cluster mode,
Service should be created by the submit binary,  in this file: org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
Don’t you have any failing logs where spark submit has been launched?

JL
From: "Prudhvi Chennuru (CONT)" <prudhvi.chennuru@capitalone.com>
Date: Tuesday 18 June 2019 at 16:15
To: Jose Luis Pedrosa <Jose.Pedrosa@microsoft.com>
Cc: Olivier Girardot <o.girardot@lateral-thoughts.com>, Li Gao <ligao101@gmail.com>,
dev <dev@spark.apache.org>, user <user@spark.apache.org>
Subject: Re: [External Sender] Re: Spark 2.4.1 on Kubernetes - DNS resolution of driver fails

Thanks for the response Oliver.

I am facing this issue intermittently, once in a while i don't see service being created for
the respective spark driver( i don't see service for that driver on kubernetes dashboard and
not even via kubectl but in driver logs i see the service endpoint) and by default driver
requests for executors in a batch of 5 as soon as 5 executors are created they fail with below
error.


Caused by: java.io.IOException: Failed to connect to group9990-features-282526d440ab3f12a68746fbef289c95-driver-svc.experimental.svc:7078
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: group9990-features-282526d440ab3f12a68746fbef289c95-driver-svc.experimental.svc
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)

Did you face the same problem or were you able to see the service for the driver pod on your
cluster?


On Tue, Jun 18, 2019 at 8:00 AM Jose Luis Pedrosa <Jose.Pedrosa@microsoft.com<mailto:Jose.Pedrosa@microsoft.com>>
wrote:
Hi guys

There’s also an interesting one that we found in a similar case. In our case the service
ip ranges takes more time to be reachable, so DNS was timing out. The approach that I was
suggesting was:

  1.  Add retries in the connection from the executor to the driver: https://github.com/apache/spark/pull/24702<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__github.com_apache_spark_pull_24702%26d%3DDwMGaQ%26c%3DpLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE%26r%3DZbC2jt41phJyXtl9lDl7uaUnDWK7Ilns1DeTPpSa2T4%26m%3Deg0fGctzE8h6HioRMam_Q18QTLAN3LEl1SdiGuTX7a4%26s%3DGA-PO2FbDWQPNYgoTNs0kNHbjryZZ6phLPZ-wdQSBTs%26e%3D&data=02%7C01%7CJose.Pedrosa%40microsoft.com%7C69d4967c1e114649b81a08d6f3ffbc85%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636964677055686109&sdata=OhTVkyXXqDoAXKV%2BUhkKgfILbjWIztodkjxTx%2F%2BBdtE%3D&reserved=0>
  2.  Disable negative DNS caching at JVM level, on the entrypoint.sh

JL


From: Olivier Girardot <o.girardot@lateral-thoughts.com<mailto:o.girardot@lateral-thoughts.com>>
Date: Tuesday 18 June 2019 at 10:06
To: "Prudhvi Chennuru (CONT)" <prudhvi.chennuru@capitalone.com<mailto:prudhvi.chennuru@capitalone.com>>
Cc: Li Gao <ligao101@gmail.com<mailto:ligao101@gmail.com>>, dev <dev@spark.apache.org<mailto:dev@spark.apache.org>>,
user <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: [External Sender] Re: Spark 2.4.1 on Kubernetes - DNS resolution of driver fails

Hi Prudhvi,
not really but we took a drastic approach mitigating this, modifying the bundled launch script
to be more resilient.
In the kubernetes/dockerfiles/spark/entrypoint.sh in the executor case we added something
like that :


  executor)

    DRIVER_HOST=$(echo $SPARK_DRIVER_URL | cut -d "@" -f 2 | cut -d ":" -f 1)

    DRIVER_PORT=$(echo $SPARK_DRIVER_URL | cut -d "@" -f 2 | cut -d ":" -f 2)



    for i in $(seq 1 20);

    do

      nc -zvw1 $DRIVER_HOST $DRIVER_PORT

      status=$?

      if [ $status -eq 0 ]

      then

        echo "Driver is accessible, let's rock'n'roll."

        break

      else

        echo "Driver not accessible :-| napping for a while..."

        sleep 3

      fi

    done

    CMD=(

      ${JAVA_HOME}/bin/java

    ....


That way the executor will not start before the driver is really connectable.
That's kind of a hack but we did not experience the issue anymore, so I guess I'll keep it
for now.

Regards,

Olivier.

Le mar. 11 juin 2019 à 18:23, Prudhvi Chennuru (CONT) <prudhvi.chennuru@capitalone.com<mailto:prudhvi.chennuru@capitalone.com>>
a écrit :
Hey Oliver,

                     I am also facing the same issue on my kubernetes cluster(v1.11.5)  on
AWS with spark version 2.3.3, any luck in figuring out the root cause?

On Fri, May 3, 2019 at 5:37 AM Olivier Girardot <o.girardot@lateral-thoughts.com<mailto:o.girardot@lateral-thoughts.com>>
wrote:
Hi,
I did not try on another vendor, so I can't say if it's only related to gke, and no, I did
not notice anything on the kubelet or kube-dns processes...

Regards

Le ven. 3 mai 2019 à 03:05, Li Gao <ligao101@gmail.com<mailto:ligao101@gmail.com>>
a écrit :
hi Olivier,

This seems a GKE specific issue? have you tried on other vendors ? Also on the kubelet nodes
did you notice any pressure on the DNS side?

Li


On Mon, Apr 29, 2019, 5:43 AM Olivier Girardot <o.girardot@lateral-thoughts.com<mailto:o.girardot@lateral-thoughts.com>>
wrote:
Hi everyone,
I have ~300 spark job on Kubernetes (GKE) using the cluster auto-scaler, and sometimes while
running these jobs a pretty bad thing happens, the driver (in cluster mode) gets scheduled
on Kubernetes and launches many executor pods.
So far so good, but the k8s "Service" associated to the driver does not seem to be propagated
in terms of DNS resolution so all the executor fails with a "spark-application-......cluster.svc.local"
does not exists.

All executors failing the driver should be failing too, but it considers that it's a "pending"
initial allocation and stay stuck forever in a loop of "Initial job has not accepted any resources,
please check Cluster UI"

Has anyone else observed this king of behaviour ?
We had it on 2.3.1 and I upgraded to 2.4.1 but this issue still seems to exist even after
the "big refactoring" in the kubernetes cluster scheduler backend.

I can work on a fix / workaround but I'd like to check with you the proper way forward :

  *   Some processes (like the airflow helm recipe) rely on a "sleep 30s" before launching
the dependent pods (that could be added to /opt/entrypoint.sh used in the kubernetes packing)
  *   We can add a simple step to the init container trying to do the DNS resolution and failing
after 60s if it did not work
But these steps won't change the fact that the driver will stay stuck thinking we're still
in the case of the Initial allocation delay.

Thoughts ?

--
Olivier Girardot
o.girardot@lateral-thoughts.com<mailto:o.girardot@lateral-thoughts.com>


--
Thanks,
Prudhvi Chennuru.

________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.


--
Olivier Girardot | Associé
o.girardot@lateral-thoughts.com<mailto:o.girardot@lateral-thoughts.com>
+33 6 24 09 17 94


--
Thanks,
Prudhvi Chennuru.

________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
View raw message