spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <java8...@hotmail.com>
Subject Re: Outer Explode needed
Date Tue, 26 Jul 2016 13:25:51 GMT
The reason of no response is that this feature is not available yet.


You can vote and following this JIRA https://issues.apache.org/jira/browse/SPARK-13721, if
you really need this feature.


Yong


________________________________
From: Don Drake <dondrake@gmail.com>
Sent: Monday, July 25, 2016 9:12 PM
To: dev@spark.apache.org
Subject: Fwd: Outer Explode needed

No response on the Users list, I thought I would repost here.

See below.

-Don
---------- Forwarded message ----------
From: Don Drake <dondrake@gmail.com<mailto:dondrake@gmail.com>>
Date: Sun, Jul 24, 2016 at 2:18 PM
Subject: Outer Explode needed
To: user <user@spark.apache.org<mailto:user@spark.apache.org>>


I have a nested data structure (array of structures) that I'm using the DSL df.explode() API
to flatten the data.  However, when the array is empty, I'm not getting the rest of the row
in my output as it is skipped.

This is the intended behavior, and Hive supports a SQL "OUTER explode()" to generate the row
when the explode would not yield any output.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

Can we get this same outer explode in the DSL?  I have to jump through some outer join hoops
to get the rows where the array is empty.

Thanks.

-Don

--
Donald Drake
Drake Consulting
http://www.drakeconsulting.com/
https://twitter.com/dondrake<http://www.MailLaunder.com/>
800-733-2143<tel:800-733-2143>



--
Donald Drake
Drake Consulting
http://www.drakeconsulting.com/
https://twitter.com/dondrake<http://www.MailLaunder.com/>
800-733-2143

Mime
View raw message