spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Night Wolf <nightwolf...@gmail.com>
Subject Re: Spark SQL ThriftServer Impersonation Support
Date Sun, 03 May 2015 12:55:19 GMT
Thanks Andrew. What version of HS2 is the SparkSQL thrift server using?
What would be involved in updating? Is it a simple case of increasing the
deep version in one of the project POMs?

Cheers,
~N

On Sat, May 2, 2015 at 11:38 AM, Andrew Lee <alee526@hotmail.com> wrote:

> Hi N,
>
> See: https://issues.apache.org/jira/browse/SPARK-5159
>
> I don't think it is yet supported until the HS2 code base is updated in
> Spark hive-thriftserver project.
>
> ------------------------------
> Date: Fri, 1 May 2015 15:56:30 +1000
> Subject: Spark SQL ThriftServer Impersonation Support
> From: nightwolfzor@gmail.com
> To: user@spark.apache.org
>
>
> Hi guys,
>
>
> Trying to use the SparkSQL Thriftserver with hive metastore. It seems that
> hive meta impersonation works fine (when running Hive tasks). However
> spinning up SparkSQL thrift server, impersonation doesn't seem to work...
>
> What settings do I need to enable impersonation?
>
> I've copied the same config as in my hive-site. Here is my launch command
> for the spark thrift server;
>
> --hiveconf hive.server2.enable.impersonation=true --hiveconf
> hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
>
> Here is my full run script:
>
> export HIVE_SERVER2_THRIFT_BIND_HOST=0.0.0.0
> export HIVE_SERVER2_THRIFT_PORT=10000
>
> export HIVE_CONF_DIR=/opt/mapr/hive/hive-0.13/conf/
> export HIVE_HOME=/opt/mapr/hive/hive-0.13/
> export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.5.1/
> export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop
>
> export EXECUTOR_MEMORY=30g
> export DRIVER_MEMORY=4g
> export EXECUTOR_CORES=15
> export NUM_EXECUTORS=20
> export KRYO_BUFFER=512
> export SPARK_DRIVER_MAXRESULTSIZE=4096
>
> export HIVE_METASTORE_URIS=thrift://localhost:9083
> export HIVE_METASTORE_WAREHOUSE_DIR=/user/hive/warehouse
>
> export
> SPARK_DIST_CLASSPATH=/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*
> export SPARK_LOG_DIR=/tmp/spark-log
> export
> SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=hdfs:///log/spark-events
> export SPARK_CONF_DIR=/apps/spark/global-conf/
>
> export SPARK_HOME=/apps/spark/spark-1.3.1-bin-mapr4.0.2_yarn_j6_2.10
>
> export SPARK_LIBRARY_PATH=/opt/mapr/lib/*
> export SPARK_JAVA_OPTS=-Djava.library.path=/opt/mapr/lib
>
>
> $SPARK_HOME/*sbin/start-thriftserver.sh* --master yarn-client --jars
> /opt/mapr/lib/libjpam.so --executor-memory $EXECUTOR_MEMORY --driver-memory
> $DRIVER_MEMORY --executor-cores $EXECUTOR_CORES --num-executors
> $NUM_EXECUTORS --conf spark.scheduler.mode=FAIR --conf
> spark.kryoserializer.buffer.mb=$KRYO_BUFFER --conf
> spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
> spark.files.useFetchCache=false --conf
> spark.driver.maxResultSize=$SPARK_DRIVER_MAXRESULTSIZE --hiveconf
> hive.metastore.uris=$HIVE_METASTORE_URIS --hiveconf
> hive.metastore.warehouse.dir=$HIVE_METASTORE_WAREHOUSE_DIR --hiveconf
> hive.server2.enable.impersonation=true --hiveconf
> hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
>
>
> Cheers,
> N
>

Mime
View raw message