spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirav Patel <npa...@xactlycorp.com>
Subject CSV parser - is there a way to find malformed csv record
Date Mon, 08 Oct 2018 18:57:09 GMT
I am getting `RuntimeException: Malformed CSV record` while parsing csv
record and attaching schema at same time. Most likely there are additional
commas or json data in some field which are not escaped properly. Is there
a way CSV parser tells me which record is malformed?


This is what I am using:

    val df2 = sparkSession.read
      .option("inferSchema", true)
      .option("multiLine", true)
      .schema(headerDF.schema) // this only works without column mismatch
      .csv(dataPath)

Thanks

-- 


 <http://www.xactlycorp.com/email-click/>

 
<https://www.instagram.com/xactlycorp/>   
<https://www.linkedin.com/company/xactly-corporation>   
<https://twitter.com/Xactly>   <https://www.facebook.com/XactlyCorp>   
<http://www.youtube.com/xactlycorporation>

Mime
View raw message