spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: Spark CSV Quote only NOT NULL
Date Sat, 13 Jul 2019 19:17:21 GMT
Hi Swetha,
I always look into the  source code a lot but it never occured to me to
look into the test suite, thank a ton for the tip.  Does definitely give
quite a few ideas - thanks a ton.

Thanks and Regards,
Gourav

On Fri, Jul 12, 2019 at 6:51 AM Swetha Ramaiah <swetha.ramaiah@gmail.com>
wrote:

> Hi Anil
>
> That was an example. You can replace quote with what double quotes. But
> these options should give you an idea on how you want treat nulls, empty
> values and quotes.
>
> When I faced this issues, I forked Spark repo and looked at the test
> suite. This definitely helped me solve my issue.
>
> https://github.com/apache/spark/blob/v2.4.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
>
> Regards
> Swetha
>
> On Jul 11, 2019, at 4:09 PM, Anil Kulkarni <anil77k@gmail.com> wrote:
>
> Hi Swetha,
>
> Thank you.
> But we need the data to be quoted with ".
> and when a field is null, we dont need the quotes around it.
>
> Example:
> "A",,"B","C"
> Thanks
> Anil
>
> On Thu, Jul 11, 2019, 1:51 PM Swetha Ramaiah <swetha.ramaiah@gmail.com>
> wrote:
>
>> If you are using Spark 2.4.0, I think you can try something like this:
>>
>> .option("quote", "\u0000")
>> .option("emptyValue", “”)
>>
>> .option("nullValue", null)
>>
>> Regards
>> Swetha
>>
>>
>>
>> On Jul 11, 2019, at 1:45 PM, Anil Kulkarni <anil77k@gmail.com> wrote:
>>
>> Hi Spark users,
>>
>> My question is :
>> I am writing a Dataframe to csv.
>> Option i am using as
>> .option("quoteAll","true").
>>
>> This is quoting even null values and making them appear as an empty
>> string.
>>
>> How do i make sure that quotes are enabled only for non null values?
>>
>> --
>> Cheers,
>> Anil Kulkarni
>> about.me/anilkulkarni
>> [image: Anil Kulkarni on about.me]
>>  http://anilkulkarni.com/ <http://about.me/anilkulkarni>
>>
>>
>>
>

Mime
View raw message