spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Herman Yu <herman...@teeupdata.com>
Subject Re: access spark thrift server from another spark session
Date Tue, 04 Oct 2016 13:34:06 GMT
Yes, I did set spark.sql.hive.thriftServer.singleSession to true in spark-defaults.conf of
both spark sessions. after starting the 2nd spark session, I manually set hive.server2.thrift.port
to the spark thrift port started within the 1st spark session, the temporary table is still
not visible.


The temp table is visible when I connect to the spark thrift server using beeline ( expecting
that it should also work with other JDBC clients even though I didn’t test), I am just not
able to make it available to another spark session, or, is it possible?


Thanks
Herman


> On Oct 3, 2016, at 22:02, ayan guha <guha.ayan@gmail.com> wrote:
> 
> I do not think you can see temp tables from other application, like thrift. You need
save the tables in hive and then they will be visible through thrift. Thrift uses hive metastore.
However temp tables do not make to central metastore until saved.
> 
> On 4 Oct 2016 11:44, "Takeshi Yamamuro" <linguin.m.s@gmail.com <mailto:linguin.m.s@gmail.com>>
wrote:
> -dev +user
> 
> Hi,
> 
> Have you try to share a session by `spark.sql.hive.thriftServer.singleSession`?
> 
> // maropu
> 
> On Tue, Oct 4, 2016 at 6:10 AM, Herman Yu <herman.yu@teeupdata.com <mailto:herman.yu@teeupdata.com>>
wrote:
> 
> I built spark data frame/dataset on top of several hive tables, and then registered dataframe/dataset
as temporary tables, as well as exposed the temporary table through spark thrift server. Now
the question is, this temporary table is only visible to the same spark session. How do I
make the registered temporary table available to another spark session?
> 
> I have tried manually set the hive.server2.thrift.port in the new spark session,but the
temporary table is still not seen. I also tried connecting to thrift server through JDBC from
the 2nd spark session, it doesn’t work either (seems spark doesn’t like the idea accessing
hive through JDBC).
> 
> so the questions is: is it possible to share data frame/dataset based temporary tables
through Spark thrift server between multiple spark sessions?
> 
> Thanks
> Herman.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <mailto:dev-unsubscribe@spark.apache.org>
> 
> 
> 
> 
> -- 
> ---
> Takeshi Yamamuro


Mime
View raw message