spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From epettijohn <vienn...@gmail.com>
Subject Re: How to control saveAsTable() warehouse path?
Date Thu, 15 Dec 2016 22:57:47 GMT
I don't profess to be an expert on this, but I did face the same problem.  A
couple of possibilities:

1.  If your default Hive database is stored in "/tmp/hive/warehouse/...",
then that could be the issue.  I recommend creating a database on s3a and
then storing the table there ( .saveAsTable('s3db.test_table',... )

2.  Have you tried setting spark.sql.warehouse.dir?  I know the docs say
that it uses the hive metastore dir first, but I've had some luck with
setting that one.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-control-saveAsTable-warehouse-path-tp23340p28221.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message