I see. Since all Spark SQL queries must be issued from the driver side, you'll have to first collect all interested values to the driver side, and then use them to compose one or more insert statements.


On 3/16/15 10:33 PM, patcharee wrote:
I would like to insert the table, and the value of the partition column to be inserted must be from temporary registered table/dataframe.


On 16. mars 2015 15:26, Cheng Lian wrote:

Not quite sure whether I understand your question properly. But if you just want to read the partition columns, it’s pretty easy. Take the “year” column as an example, you may do this in HiveQL:

hiveContext.sql("SELECT year FROM speed")

or in DataFrame DSL:



On 3/16/15 9:59 PM, patcharee wrote:


I tried to insert into a hive partitioned table

val ZONE: Int = Integer.valueOf(args(2))
val MONTH: Int = Integer.valueOf(args(3))
val YEAR: Int = Integer.valueOf(args(4))

weightedUVToDF = weightedUVToRecord.toDF()
hiveContext.sql("INSERT OVERWRITE table speed partition (year=" + YEAR + ",month=" + MONTH + ",zone=" + ZONE + ")
    select key, speed, direction from speeddata"

First I registered a temporary table "speeddata". The value of the partitioned column (year, month, zone) is from user input. If I would like to get the value of the partitioned column from the temporary table, how can I do that?