spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Stetzler <steven.stetz...@gmail.com>
Subject Problem running Spark on Kubernetes: Certificate error
Date Thu, 13 Dec 2018 21:48:51 GMT
Hello,

I am following the tutorial here (
https://spark.apache.org/docs/latest/running-on-kubernetes.html) to get
spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with
Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG
environment variable to point to my cluster access credentials, so both
Spark and kubectl can speak with the nodes.

I am running into an issue when trying to run the SparkPi example as
described in the Spark on Kubernetes tutorials. The command I am running
is:

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name
spark-pi --class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=1 --conf
spark.kubernetes.container.image=$IMAGEURL --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL
of the Spark docker image I am using (
https://hub.docker.com/r/stevenstetzler/spark/). This docker image was
built and pushed with the script included in the Spark 2.4 distribution. I
have created a service account for Spark to ensure that it has proper
permissions to create pods etc., which I checked using

kubectl auth can-i create pods --as=system:serviceaccount:default:spark

When I try to run the SparkPi example using the above command, I get the
following output:

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to
a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0)
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind
to another address
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: N/A
         start time: N/A
         container images: N/A
         phase: Pending
         status: []
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: N/A
         container images: N/A
         phase: Pending
         status: []
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: docker.io/stevenstetzler/spark:v1
         phase: Pending
         status: [ContainerStatus(containerID=null, image=
docker.io/stevenstetzler/spark:v1, imageID=,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null, terminated=null,
waiting=ContainerStateWaiting(message=null, reason=ContainerCreating,
additionalProperties={}), additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to
finish...
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: stevenstetzler/spark:v1
         phase: Running
         status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=true,
restartCount=0,
state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z,
additionalProperties={}), additionalProperties={}), terminated=null,
waiting=null, additionalProperties={}), additionalProperties={})]
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed,
new state:
         pod name: spark-pi-1544595975520-driver
         namespace: default
         labels: spark-app-selector ->
spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2
         creation time: 2018-12-12T06:26:18Z
         service account name: spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt
         node name: flamboyant-darwin-3rhc
         start time: 2018-12-12T06:26:18Z
         container images: stevenstetzler/spark:v1
         phase: Failed
         status:
[ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
image=stevenstetzler/spark:v1,
imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041,
lastState=ContainerState(running=null, terminated=null, waiting=null,
additionalProperties={}), name=spark-kubernetes-driver, ready=false,
restartCount=0, state=ContainerState(running=null,
terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a,
exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z,
additionalProperties={}), message=null, reason=Error, signal=null,
startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}),
additionalProperties={}), waiting=null, additionalProperties={}),
additionalProperties={})]
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final
statuses:


         Container name: spark-kubernetes-driver
         Container image: stevenstetzler/spark:v1
         Container state: Terminated
         Exit code: 1
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished.
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory
/tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd

When I check the logs with

kubectl logs spark-pi-1544595975520-driver

I get the following:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf
"spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
"$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file
/opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi
spark-internal
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to:
root
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups
to:
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups
to:
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager:
authentication disabled; ui acls disabled; users  with view permissions:
Set(root); groups with view permissions: Set(); users  with modify
permissions: Set(root); groups with modify permissions: Set()
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service
'sparkDriver' on port 7078.
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 -
BlockManagerMasterEndpoint up
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with
capacity 413.9 MB
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build
timestamp: unknown, git hash: unknown
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started
ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI'
on port 4040.
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6b410923
{/jobs/job/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1d3e6d34
{/stages/stage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@26a94fa5
{/stages/pool/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@577f9109
{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@757529a4
{/environment/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5c41d037
{/executors/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2234078
{/executors/threadDump,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5ec77191
{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@534ca02b
{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@29a23c3d
{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and
started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR
file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at
spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar
with timestamp 1544595983645
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing
SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
        at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
        at scala.Option.getOrElse(Option.scala:121)
        at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
        at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
        at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
        ... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
        at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
        ... 23 more
Caused by: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
        ... 29 more
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701
{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at
http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 -
MapOutputTrackerMasterEndpoint stopped!
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster
stopped
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that
is not running
2018-12-12 06:26:23 INFO
 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped
SparkContext
Exception in thread "main" org.apache.spark.SparkException: External
scheduler cannot be instantiated
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
        at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
        at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
        at scala.Option.getOrElse(Option.scala:121)
        at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
has occurred.
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
        at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167)
        at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
        at
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
        at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
        ... 20 more
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
        at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
        at
io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
        at
io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
        at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
        ... 23 more
Caused by: java.io.IOException: Empty input
        at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
        ... 29 more
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
/var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory
/tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa

It looks to me like there is a communication / authentication issue during
pod creation, and it looks like there might be a certificate missing that
Spark needs to communicate with Kubernetes, which I infer from the
"java.security.cert.CertificateException: Could not parse certificate:
java.io.IOException: Empty input" error message.

Any ideas on what I can do to resolve this? Thanks!

Mime
View raw message