spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HARSH TAKKAR <takkarha...@gmail.com>
Subject Re: Hive External Table Partiton Data Type.
Date Mon, 16 Dec 2019 08:13:23 GMT
Hi,


I am able to make it in spark 2.3.0, If you can change the version to spark
2.3 it will be good , otherwise let me know , i 'll check on the Spark
version 2.1.0.

Following is the code for spark 2.3.0.

scala> var seq = Seq((10L,"Hello"),(10L,"Hi"))
seq: Seq[(Long, String)] = List((10,Hello), (10,Hi))

scala> seq.toDF("a","b")
res0: org.apache.spark.sql.DataFrame = [a: bigint, b: string]

scala> res0.show()
+---+-----+
|  a|    b|
+---+-----+
| 10|Hello|
| 10|   Hi|
+---+-----+

scala> res0.printSchema
root
 |-- a: long (nullable = false)
 |-- b: string (nullable = true)


scala>
res0.write.option("path","/tmp/longpartition1").mode("overwrite").partitionBy("a").saveAsTable("default.longPartition")


hive> select * from longpartition
    > ;
OK
Hello 10
Hi 10
Time taken: 0.356 seconds, Fetched: 2 row(s)
hive> describe longpartition;
OK
b                   string
a                   bigint

# Partition Information
# col_name             data_type           comment

a                   bigint

On Mon, Dec 16, 2019 at 11:05 AM SB M <manjusree1062@gmail.com> wrote:

> spark version 2.1.0
>
> Regards,
> Sbm
>
> On Mon, 16 Dec, 2019, 10:04 HARSH TAKKAR, <takkarharsh@gmail.com> wrote:
>
>> Please share the spark version you are using .
>>
>> On Fri, 13 Dec, 2019, 4:02 PM SB M, <manjusree1062@gmail.com> wrote:
>>
>>> Hi All,
>>>    Am trying to create a dynamic partition with external table on hive
>>> metastore using spark sql.
>>>
>>> when am trying to create a partition column data type as bigint,
>>> partition is not working even i tried with repair table. data is not shown
>>> when i ran sample query select * from table.
>>>
>>>
>>> but if i tried to create a dynamic partition with string as data type
>>> for partition its working fine. partition are working as expected.
>>>
>>>
>>> is there something am doing it wrong ??
>>>
>>>
>>> Regards,
>>> Sree
>>>
>>>

Mime
View raw message