spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Kravchenko <kravchenko.anto...@gmail.com>
Subject spark reshape hive table and save to parquet
Date Thu, 08 Dec 2016 16:53:25 GMT
Hello,

I wonder if there is a way (preferably efficient) in Spark to reshape hive
table and save it to parquet.

Here is a minimal example, input hive table:
col1 col2 col3
1 2 3
4 5 6

output parquet:
col1 newcol2
1 [2 3]
4 [5 6]

p.s. The real input hive table has ~1000 columns.

Thank you,
Anton

Mime
View raw message