spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Convert a line of String into column
Date Tue, 01 Oct 2019 21:48:26 GMT
I want to convert a line of String to a table. For instance, I want to convert following line

<column1> <column2> <columns> ...<column6> # this is a line in a text
file, separated by a white space

to table 

|col1| col2| col3...|col6|

The code looks as below

    import org.apache.spark.sql.functions._    import org.apache.spark.sql.SparkSession  
 val spark = SparkSession      .builder      .master("local")      .appName("MyApp")     
.getOrCreate()    import spark.implicits._    val lines = spark.readStream.textFile("/tmp/data/")
   val words =[String].flatMap(_.split(" "))    words.printSchema()    val query
= words.      writeStream.      outputMode("append").      format("console").      start 
But in fact this code only turns the line into a single column

| value|
| col3..|
|  ...     |
|  col6 |

How to achieve the effect that I want to do?


View raw message