spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kalvin Chau <work.kalvinnc...@gmail.com>
Subject Spark Authentication on Mesos Client Mode
Date Tue, 22 Aug 2017 22:17:55 GMT
Hi,

Is it possible to use spark.authenticate with shared secrets in mesos
client mode? I can get the driver to start up and expect to use
authenticated channels, but when the executors start up the SecurityManager
outputs "authentication: disabled".

Looking through the CoarseGrainedSchelduerBackend code, it just creates a
new SparkEnv (and doesn't look like it loads spark-defaults.conf) and
attempts to fetch properties from the driver.
But the driver is expecting an authentication challenge and bails when it
realizes the executor is not attempting the challenge.

Gist of some of the logs
https://gist.github.com/kalvinnchau/53c60c91a0e1b47ad0afd9fe06d81101

Relevant code here, I don't see where the executor would load up auth
settings and grab the shared secret.

      // Bootstrap to fetch the driver's Spark properties.
      val executorConf = new SparkConf
      val port = executorConf.getInt("spark.executor.port", 0)
      val fetcher = RpcEnv.create(
        "driverPropsFetcher",
        hostname,
        port,
        executorConf,
        new SecurityManager(executorConf),
        clientMode = true)
      val driver = fetcher.setupEndpointRefByURI(driverUrl)
      val cfg = driver.askSync[SparkAppConfig](RetrieveSparkAppConfig)
      val props = cfg.sparkProperties ++ Seq[(String, String)](("
spark.app.id", appId))
      fetcher.shutdown()


Is there some setting I'm missing to get the executor backend to pull in
authentication settings?

Thanks,
Kalvin

Mime
View raw message