spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Stetzler <steven.stetz...@gmail.com>
Subject Re: Problem running Spark on Kubernetes: Certificate error
Date Thu, 20 Dec 2018 15:13:41 GMT
Hi Matt,

Thank your for the help. This worked for me. For posterity: convert the
data in the certificate-authority-data field of your DigitalOcean
Kubernetes configuration file, which is downloaded from their site, from
base64 to PEM format and save it to a file ca.crt, then submit with --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt.

As an extension to this question, I am also interested in using PySpark on
a JupyterHub deployment to interact with my cluster. The JupyterHub
deployment provides a Python (with PySpark) environment in a pod on a node
in the cluster, and I would like to create a SparkContext for computations
that uses the same cluster. I have attempted to replicate the above
solution there using:

from pyspark import *
> conf = SparkConf()
> conf.setMaster('k8s://<CLUSTERIP>')
> conf.set('spark.kubernetes.container.image', '<IMAGEURL>')
> conf.set('spark.kubernetes.authenticate.driver.caCertFile', 'ca.crt')
> sc = SparkContext(conf=conf)


which returns an identical error:

---------------------------------------------------------------------------
> Py4JJavaError Traceback (most recent call last)
> <ipython-input-4-bfc2d0364053> in <module>
> 22
> 23
> ---> 24 sc = SparkContext(conf=conf)
> /usr/local/spark/python/pyspark/context.py in __init__(self, master,
> appName, sparkHome, pyFiles, environment, batchSize, serializer, conf,
> gateway, jsc, profiler_cls)
> 116 try:
> 117 self._do_init(master, appName, sparkHome, pyFiles, environment,
> batchSize, serializer,
> --> 118 conf, jsc, profiler_cls) 119 except:
> 120 # If an error occurs, clean up in order to allow future SparkContext
> creation: /usr/local/spark/python/pyspark/context.py in _do_init(self,
> master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
> conf, jsc, profiler_cls)
> 178
> 179 # Create the Java SparkContext through Py4J
> --> 180 self._jsc = jsc or self._initialize_context(self._conf._jconf)
> 181 # Reset the SparkConf to the one actually used by the SparkContext in
> JVM.
> 182 self._conf = SparkConf(_jconf=self._jsc.sc().conf())
> /usr/local/spark/python/pyspark/context.py in _initialize_context(self,
> jconf)
> 286 Initialize SparkContext in function to allow subclass specific
> initialization
> 287 """
> --> 288 return self._jvm.JavaSparkContext(jconf)
> 289
> 290 @classmethod
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in
> __call__(self, *args)
> 1523 answer = self._gateway_client.send_command(command)
> 1524 return_value = get_return_value(
> -> 1525 answer, self._gateway_client, None, self._fqn) 1526
> 1527 for temp_arg in temp_args:
> /usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in
> get_return_value(answer, gateway_client, target_id, name)
> 326 raise Py4JJavaError(
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value) 329 else:
> 330 raise Py4JError( Py4JJavaError: An error occurred while calling
> None.org.apache.spark.api.java.JavaSparkContext.
> : org.apache.spark.SparkException: External scheduler cannot be
> instantiated
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:238)
> at
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> ... 13 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
> ... 16 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 22 more


On the other hand, I have no problem running

from pyspark import *
> sc = SparkContext()


and then performing a computation. However I believe that this will only be
able to use the resources provided to the Kubernetes pod that the Jupyter
environment is deployed on and not the whole cluster, so it's not really
what I'm looking for. I have tried running the same Python code from
outside of the cluster (on my own machine with PySpark installed). Note
that it doesn't matter what IP I use for the master, the same error occurs.
Running from outside of the cluster does seem to allow PySpark to
communicate with the cluster. However, there appear to be issues allocating
resources / creating workers. (But that seems like an issue for another
thread.)

Any idea what can be done here? Note that if it was really passing the file
and the file was not found, it would raise an error to indicate that. (I
have tried.) The error raised indicates instead that no file was ever
passed. I would assume that
conf.set('spark.kubernetes.authenticate.driver.caCertFile',
'ca.crt') would have identical behavior to --conf
spark.kubernetes.authenticate.driver.caCertFile=ca.crt in spark-submit.
Sorry if I am misunderstanding something basic in how PySpark and
spark-submit work.

Thanks,
Steven


On Thu, Dec 13, 2018 at 7:59 PM Matt Cheah <mcheah@palantir.com> wrote:

> Hi Steven,
>
>
>
> What I think is happening is that your machine has a CA certificate that
> is used for communicating with your API server, particularly because you’re
> using Digital Ocean’s cluster manager. However, it’s unclear if your pod
> has the same CA certificate or if the pod needs that certificate file. You
> can use configurations to have your pod use a particular CA certificate
> file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile
> to the path of your CA certificate on your local disk, spark-submit will
> create a secret that contains that certificate file and use that
> certificate to configure TLS for the driver pod’s communication with the
> API server.
>
>
>
> It's clear that your driver pod doesn’t have the right TLS certificate to
> communicate with the API server, so I would try to introspect the driver
> pod and see what certificate it’s using for that communication. If there’s
> a fix that needs to happen in Spark, feel free to indicate as such.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Steven Stetzler <steven.stetzler@gmail.com>
> *Date: *Thursday, December 13, 2018 at 1:49 PM
> *To: *"user@spark.apache.org" <user@spark.apache.org>
> *Subject: *Problem running Spark on Kubernetes: Certificate error
>
>
>
> Hello,
>
> I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html
> [spark.apache.org]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=9gQXKMnjixgZA7jGBSWFYdhjnBsgzdy51B5bZJ5znQ8&e=>)
> to get spark running on a Kubernetes cluster. My Kubernetes cluster is
> hosted with Digital Ocean's kubernetes cluster manager. I have change the
> KUBECONFIG environment variable to point to my cluster access credentials,
> so both Spark and kubectl can speak with the nodes.
>
> I am running into an issue when trying to run the SparkPi example as
> described in the Spark on Kubernetes tutorials. The command I am running
> is:
>
> ./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
> spark-pi --class org.apache.spark.examples.SparkPi --conf
> spark.executor.instances=1 --conf
> spark.kubernetes.container.image=$IMAGEURL --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark
> local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
>
> where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the
> URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/
> [hub.docker.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__hub.docker.com_r_stevenstetzler_spark_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=lgzK2iTO0oHaHnD6w81b-Cwl8YAAK5XyddxSBCi35j0&e=>).
> This docker image was built and pushed with the script included in the
> Spark 2.4 distribution. I have created a service account for Spark to
> ensure that it has proper permissions to create pods etc., which I checked
> using
>
> kubectl auth can-i create pods --as=system:serviceaccount:default:spark
>
> When I try to run the SparkPi example using the above command, I get the
> following output:
>
> 2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves
> to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface
> eth0)
> 2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to
> bind to another address
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: N/A
>          start time: N/A
>          container images: N/A
>          phase: Pending
>          status: []
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: N/A
>          container images: N/A
>          phase: Pending
>          status: []
> 2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: docker.io/stevenstetzler/spark:v1 [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>
>
>          phase: Pending
>          status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1
> [docker.io]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.io_stevenstetzler_spark-3Av1&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=O4l8wZMrO52IV576ItzHzN_REr0NCrRNnZ-0ZCB30a0&e=>,
> imageID=, lastState=ContainerState(running=null, terminated=null,
> waiting=null, additionalProperties={}), name=spark-kubernetes-driver,
> ready=false, restartCount=0, state=ContainerState(running=null,
> terminated=null, waiting=ContainerStateWaiting(message=null,
> reason=ContainerCreating, additionalProperties={}),
> additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to
> finish...
> 2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: stevenstetzler/spark:v1
>          phase: Running
>          status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=true,
> restartCount=0,
> state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
> additionalProperties={}), additionalProperties={}), terminated=null,
> waiting=null, additionalProperties={}), additionalProperties={})]
> 2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
> new state:
>          pod name: spark-pi-1544595975520-driver
>          namespace: default
>          labels: spark-app-selector ->
> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
>          pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
>          creation time: 2018-12-12T06:26:18Z
>          service account name: spark
>          volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
>          node name: flamboyant-darwin-3rhc
>          start time: 2018-12-12T06:26:18Z
>          container images: stevenstetzler/spark:v1
>          phase: Failed
>          status:
> [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> image=stevenstetzler/spark:v1,
> imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
> lastState=ContainerState(running=null, terminated=null, waiting=null,
> additionalProperties={}), name=spark-kubernetes-driver, ready=false,
> restartCount=0, state=ContainerState(running=null,
> terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
> exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
> additionalProperties={}), message=null, reason=Error, signal=null,
> startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
> additionalProperties={}), waiting=null, additionalProperties={}),
> additionalProperties={})]
> 2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final
> statuses:
>
>
>          Container name: spark-kubernetes-driver
>          Container image: stevenstetzler/spark:v1
>          Container state: Terminated
>          Exit code: 1
> 2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished.
> 2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory
> /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd
>
> When I check the logs with
>
> kubectl logs spark-pi-1544595975520-driver
>
> I get the following:
>
> ++ id -u
> + myuid=0
> ++ id -g
> + mygid=0
> + set +e
> ++ getent passwd 0
> + uidentry=root:x:0:0:root:/root:/bin/ash
> + set -e
> + '[' -z root:x:0:0:root:/root:/bin/ash ']'
> + SPARK_K8S_CMD=driver
> + case "$SPARK_K8S_CMD" in
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sed 's/[^=]*=\(.*\)/\1/g'
> + sort -t_ -k4 -n
> + readarray -t SPARK_EXECUTOR_JAVA_OPTS
> + '[' -n '' ']'
> + '[' -n '' ']'
> + PYSPARK_ARGS=
> + '[' -n '' ']'
> + R_ARGS=
> + '[' -n '' ']'
> + '[' '' == 2 ']'
> + '[' '' == 3 ']'
> + case "$SPARK_K8S_CMD" in
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
> /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
> spark-internal
> 2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0
> 2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark
> Pi
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to:
> root
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups
> to:
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups
> to:
> 2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users  with view permissions:
> Set(root); groups with view permissions: Set(); users  with modify
> permissions: Set(root); groups with modify permissions: Set()
> 2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
> 'sparkDriver' on port 7078.
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster
> 2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
> 2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with
> capacity 413.9 MB
> 2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering
> OutputCommitCoordinator
> 2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms
> 2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build
> timestamp: unknown, git hash: unknown
> 2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms
> 2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started
> ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
> 'SparkUI' on port 4040.
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b410923
> {/jobs/job/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@203dd56b
> {/stages/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@e8fadb0
> {/stages/stage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1d3e6d34
> {/stages/stage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6eafb10e
> {/stages/pool,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@26a94fa5
> {/stages/pool/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2873d672
> {/storage/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3bc735b3
> {/storage/rdd,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@577f9109
> {/storage/rdd/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4303b7f0
> {/environment,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@757529a4
> {/environment/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5c41d037
> {/executors/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2234078
> {/executors/threadDump,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@5ec77191
> {/executors/threadDump/json,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@534ca02b
> {/jobs/job/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@29a23c3d
> {/stages/stage/kill,null,AVAILABLE,@Spark}
> 2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and
> started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR
> file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
> spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
> with timestamp 1544595983645
> 2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
> SparkContext.
> org.apache.spark.SparkException: External scheduler cannot be instantiated
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
>         at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
>         at scala.Option.getOrElse(Option.scala:121)
>         at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
>         at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
>         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>         at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>         at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>         at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>         at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>         at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
>         at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
>         at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
>         ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
>         at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
>         ... 23 more
> Caused by: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
>         ... 29 more
> 2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701
> {HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0-3A4040&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=NEpJ4w980MUskRPIbAZm04Qx67LafCpnymbJwyzat4Y&e=>
> }
> 2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040
> [spark-pi-1544595975520-driver-svc.default.svc]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark-2Dpi-2D1544595975520-2Ddriver-2Dsvc.default.svc-3A4040_&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=7_pmhkpOYkR7IRMQolPOXsfzMG9byW5WEPVrfs4UgCg&e=>
> 2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 -
> MapOutputTrackerMasterEndpoint stopped!
> 2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared
> 2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped
> 2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster
> stopped
> 2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that
> is not running
> 2018-12-12 06:26:23 INFO
>  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
> OutputCommitCoordinator stopped!
> 2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped
> SparkContext
> Exception in thread "main" org.apache.spark.SparkException: External
> scheduler cannot be instantiated
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
>         at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
>         at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
>         at scala.Option.getOrElse(Option.scala:121)
>         at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
>         at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
>         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org
> [org.apache.spark.deploy.sparksubmit.org]
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.deploy.SparkSubmit.org&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=koBmBBRy1RJd_Z00W1RO9fi0a72z2vYhFoj-O0FN3RQ&s=UFf1bgM3AcQSaqrUz94wpKROIXo4lHTVmIa6KAOv2vo&e=>
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>         at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>         at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>         at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
>         at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
>         at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An
> error has occurred.
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
>         at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
>         at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
>         at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
>         at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
>         ... 20 more
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
>         at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
>         at
> io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
>         at
> io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
>         at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
>         ... 23 more
> Caused by: java.io.IOException: Empty input
>         at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
>         ... 29 more
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
> /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
> 2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
> /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa
>
> It looks to me like there is a communication / authentication issue during
> pod creation, and it looks like there might be a certificate missing that
> Spark needs to communicate with Kubernetes, which I infer from the
> "java.security.cert.CertificateException: Could not parse certificate:
> java.io.IOException: Empty input" error message.
>
> Any ideas on what I can do to resolve this? Thanks!
>

Mime
View raw message