spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From suresh kumar pathak <pathak.sur...@gmail.com>
Subject Re: How can transform RDD[Seq[String]] to RDD[ROW]
Date Thu, 05 Aug 2021 07:41:34 GMT
May be this link will help you.
https://stackoverflow.com/questions/41898144/convert-rddstring-to-rddrow-to-dataframe-spark-scala

On Thu, Aug 5, 2021 at 12:46 PM igyu <igyu@21cn.com> wrote:

> val ftpUrl = "ftp://test:test@ip:21/upload/test/_temporary/0/_temporary/task_20191211114756_0002_m_000000_0/*"
> val rdd = spark.sparkContext.wholeTextFiles(ftpUrl)
> val value = rdd.map(_._2).map(csv=>csv.split(",").toSeq)
>
> val schemas = StructType(List(
>         new StructField("id", DataTypes.StringType, true),
>         new StructField("name", DataTypes.StringType, true),
>         new StructField("year", DataTypes.IntegerType, true),
>         new StructField("city", DataTypes.StringType, true)))
> val DF = spark.createDataFrame(value,schemas)
>
> How can I createDataFrame
>
> ------------------------------
> igyu
>


-- 
Thanks & Regards,
Suresh Kumar Pathak(+918884772233)

Mime
View raw message