Hi Divya,

Thanks, it is exactly what I am looking for!

Anton

On Wed, Dec 14, 2016 at 6:01 PM, Divya Gehlot <divya.htconex@gmail.com> wrote:
you can use udfs to do it 
http://stackoverflow.com/questions/31615657/how-to-add-a-new-struct-column-to-a-dataframe

Hope it will help.


Thanks,
Divya

On 9 December 2016 at 00:53, Anton Kravchenko <kravchenko.anton86@gmail.com> wrote:
Hello,

I wonder if there is a way (preferably efficient) in Spark to reshape hive table and save it to parquet.

Here is a minimal example, input hive table:
col1 col2 col3
1 2 3
4 5 6

output parquet:
col1 newcol2 
1 [2 3]
4 [5 6]

p.s. The real input hive table has ~1000 columns.

Thank you,
Anton