spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <hamishberri...@tutanota.com>
Subject Convert a line of String into column
Date Tue, 01 Oct 2019 21:48:26 GMT
I want to convert a line of String to a table. For instance, I want to convert following line


<column1> <column2> <columns> ...<column6> # this is a line in a text
file, separated by a white space

to table 

+-----+------+----....+------+
|col1| col2| col3...|col6|
+-----+-----+-----....+-----+
|val1|val2|val3....|val6|
+-----+------+---.....+-----+
.....

The code looks as below

    import org.apache.spark.sql.functions._    import org.apache.spark.sql.SparkSession  
 val spark = SparkSession      .builder      .master("local")      .appName("MyApp")     
.getOrCreate()    import spark.implicits._    val lines = spark.readStream.textFile("/tmp/data/")
   val words = lines.as[String].flatMap(_.split(" "))    words.printSchema()    val query
= words.      writeStream.      outputMode("append").      format("console").      start 
  query.awaitTermination()
But in fact this code only turns the line into a single column

+-------+
| value|
+-------+
|col1...|
|col2...|
| col3..|
|  ...     |
|  col6 |
+------+

How to achieve the effect that I want to do?

Thanks? 


Mime
View raw message