spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jain, Nishit" <nja...@underarmour.com>
Subject CSV escaping not working
Date Thu, 27 Oct 2016 15:54:06 GMT
I am using spark-core version 2.0.1 with Scala 2.11. I have simple code to read a csv file
which has \ escapes.

val myDA = spark.read
      .option("quote",null)
    .schema(mySchema)
    .csv(filePath)


As per documentation \ is default escape for csv reader. But it does not work. Spark is reading
\ as part of my data. For Ex: City column in csv file is north rocks\,au . I am expecting
city column should read in code as northrocks,au. But instead spark reads it as northrocks\
and moves au to next column.

I have tried following but did not work:

  *   Explicitly defined escape .option("escape",”\\")
  *   Changed escape to | or : in file and in code
  *   I have tried using spark-csv library

Any one facing same issue? Am I missing something?

Thanks
Mime
View raw message