spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewan Leith <ewan.le...@realitymine.com>
Subject RE: Create column in nested structure?
Date Thu, 13 Aug 2015 14:54:52 GMT
Never mind me, I've found an email to this list from Raghavendra Pandey which got me what I
needed

val nestedCol = struct(df("nested2.column1"), df("nested2.column2"), df("flatcolumn"))
val df2 = df.select(df("nested1"), nestedCol as "nested2")

Thanks,
Ewan

From: Ewan Leith
Sent: 13 August 2015 15:44
To: user@spark.apache.org
Subject: Create column in nested structure?

Has anyone used withColumn (or another method) to add a column to an existing nested dataframe?

If I call:

    df.withColumn("nested.newcolumn", df("oldcolumn"))

then it just creates the new column with a "." In it's name, not under the "nested" structure.

Thanks,
Ewan

Mime
View raw message