spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwin Shankar <>
Subject Multitenancy in Spark - within/across spark context
Date Wed, 22 Oct 2014 18:47:21 GMT
Hi Spark devs/users,
One of the things we are investigating here at Netflix is if Spark would
suit us for our ETL needs, and one of requirements is multi tenancy.
I did read the official doc
<> and the book, but
I'm still not clear on certain things.

Here are my questions :
1. *Sharing spark context* : How exactly multiple users can share the
cluster using same spark
    context ? UserA wants to run AppA, UserB wants to run AppB. How do they
talk to same
    context ? How exactly are each of their jobs scheduled and run in same
    Is preemption supported in this scenario ? How are user names passed on
to the spark context ?

2. *Different spark context in YARN*: assuming I have a YARN cluster with
queues and preemption
    configured. Are there problems if executors/containers of a spark app
are preempted to allow a
    high priority spark app to execute ? Would the preempted app get stuck
or would it continue to
    make progress? How are user names passed on from spark to yarn(say I'm
using nested user
    queues feature in fair scheduler) ?

3. Sharing RDDs in 1 and 2 above ?

4. Anything else about user/job isolation ?

I know I'm asking a lot of questions. Thanks in advance :) !


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message