spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Any equivalent method lateral and explore
Date Fri, 25 Nov 2016 10:58:44 GMT
Hi,

Interesting, but I personally would opt for withColumn since it'd be
less to type (and also be consistent with ticks (')) as follows:

df.withColumn(explode('myArray) as 'arrayItem)

(Spark SQL made my SQL developer's life so easy these days :))

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Nov 22, 2016 at 11:58 PM, Michael Armbrust
<michael@databricks.com> wrote:
> Both collect_list and explode are available in the function library.
>
> The following is an example of using it:
> df.select($"*", explode($"myArray") as 'arrayItem)
>
> On Tue, Nov 22, 2016 at 2:42 PM, Mahender Sarangam
> <Mahender.BigData@outlook.com> wrote:
>>
>> Hi,
>>
>> We are converting our hive logic which is using lateral view and explode
>> functions. Is there any builtin function in scala for performing lateral
>> view explore.
>>
>>
>> Below is our  query in Hive. temparray is temp table with c0 and c1
>> columns
>>
>> SELECT id, CONCAT_WS(',', collect_list(LineID)) as LineiD
>> FROM (SELECT cast(LineID as STRING) as LineiD, cast(id as STRING) as id
>> FROM temparray LATERAL VIEW explode(`_c1`) adTable AS id) T
>> GROUP BY id;
>>
>>
>> Can any one provide pointer for string functions available in scala. We
>> would like perform operations like Collect_List, get starting index of
>> matching string.
>>
>>
>>
>> Nanu
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message