spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tianlang <tianlangstu...@aliyun.com.INVALID>
Subject Re: Spark streaming dataframe extract message to new columns
Date Tue, 13 Aug 2019 06:04:02 GMT
Hi,

Do you mean haven a colum A then you want to extract A1 and A2 from A ?

like

Cloumn A value   123456,2019-08-07

A1 value is 123456

A2 value is 2019-08-07

If that's the case you can use df.select like this

df.select(split('A)(0) as "A1", split('A)(1) as "A2")


Good Luck

在 2019/8/12 下午8:51, Gourav Sengupta 写道:
> Hi,
>
> I think that it should be possible to write a query on the streaming 
> data frame and then write the output of the query to S3 or any other 
> sink layer.
>
> Regards,
> Gourav Sengupta
>
> On Sat, Aug 10, 2019 at 9:24 AM zenglong chen <czlong.kelvin@gmail.com 
> <mailto:czlong.kelvin@gmail.com>> wrote:
>
>     How to extract some message in streaming dataframe and make a new
>     column?
>     Thanks.
>
-- 

TianlangStudio <https://edu.51cto.com/sd/aca72>

Some of the biggest lies: I will start tomorrow/Others are better than 
me/I am not good enough/I don't have time/This is the way I am 
<https://edu.51cto.com/sd/aca72>

<https://github.com/TianLangStudio>



Mime
View raw message