spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Undocumented left join constraint?
Date Fri, 27 May 2016 21:52:00 GMT
Sounds like: https://issues.apache.org/jira/browse/SPARK-15441, for which a
fix is in progress.

Please do keep reporting issues though, these are great!

Michael

On Fri, May 27, 2016 at 1:01 PM, Tim Gautier <tim.gautier@gmail.com> wrote:

> Is it truly impossible to left join a Dataset[T] on the right if T has any
> non-option fields? It seems Spark tries to create Ts with null values in
> all fields when left joining, which results in null pointer exceptions. In
> fact, I haven't found any other way to get around this issue without making
> all fields in T options. Is there any other way?
>
> Example:
>
>     case class Test(id: Int)
>     val test1 = Seq(Test(1), Test(2), Test(3)).toDS
>     val test2 = Seq(Test(2), Test(3), Test(4)).toDS
>     test1.as("t1").joinWith(test2.as("t2"), $"t1.id" === $"t2.id",
> "left_outer").show
>
>

Mime
View raw message