spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Divya Gehlot <>
Subject Spark 1.5.2 - Read custom schema from file
Date Fri, 04 Mar 2016 05:38:13 GMT
I have defined a custom schema as shown below :

val customSchema = StructType(
>     StructField("year", IntegerType, true),
>     StructField("make", StringType, true),
>     StructField("model", StringType, true),
>     StructField("comment", StringType, true),

    StructField("blank", StringType, true))

Is there any way instead of defining it spark job file I can read from file.
I am using Spark-csv to read my data file

 val df =
    .option("header", "true") // Use first line of all files as header
    .load("cars.csv")val selectedData ="year", "model")
    .option("header", "true")

View raw message