spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abel Rincón <gange...@gmail.com>
Subject Re: Spark Kerberos proxy user
Date Tue, 30 Aug 2016 10:02:03 GMT
Hi again,

Is there any open issue related?

Nowadays, we (stratio)  have a end to end  running, with a spark
distribution based in 1.6.2.

Work in progress:

- Create and share our solution documentation.
- Test Suite for all the stuff.
- Rebase our code with apache-master branch.

Regards,


2016-08-25 12:10 GMT+02:00 Abel Rincón <ganger85@gmail.com>:

> Hi devs,
>
> I'm working (at Stratio)  on use spark over mesos and standalone, with a
> kerberized HDFS
>
> We are working to solve these scenarios,
>
>
>    - We have an long term running spark sql context using concurrently by
>    many users like Thrift server called CrossData, we need access to hdfs data
>    with kerberos authorization using proxy-user method. we trust on HDFS
>    permission system, or our custom authorizer.
>
>
>    - We need load/write dataframes using datasources with HDFS
>    backend(built-in, or others)  such json, csv, parquet, orc …, so we want to
>    enable the secure access (krb)  only by configuration.
>
>
>    - We have an scenario where we want to run streaming jobs over
>    kerberized HDFS,  from W/R and  checkpointing too.
>
>
>    - We have to load every single RDD that spark core over kerberized
>    HDFS without breaking the Spark API.
>
>
>
>
> As you can see, We have a "special" requirement need to set the proxy user
> by job over the same spark context.
>
> Do you have any idea to cover it?
>
>

Mime
View raw message