spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-15383) Support appending new columns from other DataFrames
Date Wed, 18 May 2016 13:32:12 GMT


Sean Owen commented on SPARK-15383:

Does that make logical sense? the other DF may not have the same number of rows, or same partitioning,
or same ordering.
You can of course put two dataframes together how you like, but have to specify how it is
they are joined with existing operators.

> Support appending new columns from other DataFrames
> ---------------------------------------------------
>                 Key: SPARK-15383
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Lijie Xu
>            Priority: Minor
> It is common to add column(s) from other DataFrame(s) as follows. However, current *withColumn()*
function only supports appending a new column from the same DataFrame.
> {code:java}
>     val df1 = sc.makeRDD(1 to 5).toDF("a")
>     val df2 = sc.makeRDD(10 to 15).toDF("b")
>     val df = df1.withColumn("b", df2("b")) // Exception
> {code}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved attribute(s)
b#3 missing from a#1 in operator !Project [a#1,b#3 AS b#4];

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message