spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruce Robbins (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported
Date Sat, 09 Feb 2019 23:11:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764260#comment-16764260
] 

Bruce Robbins commented on SPARK-26804:
---------------------------------------

[~hipruthvi]

It seems that neither 2.3 nor 2.4 are properly handling the combination of carriage return
and linefeed (0x0d0a) at the end of the line. 

When I removed the carriage returns from your file such that each line ended only with linefeed
(aka newline), the problem went away, at least on 2.4.

> Spark sql carries newline char from last csv column when imported
> -----------------------------------------------------------------
>
>                 Key: SPARK-26804
>                 URL: https://issues.apache.org/jira/browse/SPARK-26804
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Raj
>            Priority: Major
>         Attachments: TestFile.csv, image-2019-02-04-12-09-19-210.png, image-2019-02-04-12-28-21-117.png
>
>
> I am trying to generate external sql tables in DataBricks using Spark sql query. Below
is my query. The query reads csv file and creates external table but it carries the newline
char while creating the last column. Is there a way to resolve this issue? 
>  
> %sql
> create table if not exists <<My table name>>
> using CSV
> options ("header"="true", "inferschema"="true","multiLine"="true", "escape"='"')
> location <my csv path>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message