spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <ravishankar.n...@gmail.com>
Subject Re: Spark Dataframe and HIVE
Date Fri, 09 Feb 2018 15:03:09 GMT
An update: (Sorry I missed)

When I do

passion_df.createOrReplaceTempView("sampleview")

spark.sql("create table sample table as select * from sample view")

Now, I can see table and can query as well.

So why this do work from Spark and other method discussed below is not?

Thanks



On Fri, Feb 9, 2018 at 9:49 AM, ☼ R Nair (रविशंकर नायर) <
ravishankar.nair@gmail.com> wrote:

> All,
>
> It has been three days continuously I am on this issue. Not getting any
> clue.
>
> Environment: Spark 2.2.x, all configurations are correct. hive-site.xml is
> in spark's conf.
>
> 1) Step 1: I created a data frame DF1 reading a csv file.
>
> 2) Did  manipulations on DF1. Resulting frame is passion_df.
>
> 3) passion_df.write.format("orc").saveAsTable("sampledb.passion")
>
> 4) The metastore shows the hive table., when I do "show tables" in HIVE, I
> can see table name
>
> 5) I can't select in HIVE, though I can select from SPARK as
> spark.sql("select * from sampledb.passion")
>
> Whats going on here? Please help. Why I am not seeing data from HIVE
> prompt?
> The "describe formatted " command on the table in HIVE shows he data is is
> in default warehouse location ( /user/hive/warehouse) since I set it.
>
> I am not getting any definite answer anywhere. Many suggestions and
> answers given in Stackoverflow et al.Nothing really works.
>
> So asking experts here for some light on this, thanks
>
> Best,
> Ravion
>
>
>


--

Mime
View raw message