spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Swetha Ramaiah <swetha.rama...@gmail.com>
Subject Re: Spark CSV Quote only NOT NULL
Date Sun, 14 Jul 2019 01:33:25 GMT
Glad to help!

On Sat, Jul 13, 2019 at 12:17 PM Gourav Sengupta <gourav.sengupta@gmail.com>
wrote:

> Hi Swetha,
> I always look into the  source code a lot but it never occured to me to
> look into the test suite, thank a ton for the tip.  Does definitely give
> quite a few ideas - thanks a ton.
>
> Thanks and Regards,
> Gourav
>
> On Fri, Jul 12, 2019 at 6:51 AM Swetha Ramaiah <swetha.ramaiah@gmail.com>
> wrote:
>
>> Hi Anil
>>
>> That was an example. You can replace quote with what double quotes. But
>> these options should give you an idea on how you want treat nulls, empty
>> values and quotes.
>>
>> When I faced this issues, I forked Spark repo and looked at the test
>> suite. This definitely helped me solve my issue.
>>
>> https://github.com/apache/spark/blob/v2.4.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
>>
>> Regards
>> Swetha
>>
>> On Jul 11, 2019, at 4:09 PM, Anil Kulkarni <anil77k@gmail.com> wrote:
>>
>> Hi Swetha,
>>
>> Thank you.
>> But we need the data to be quoted with ".
>> and when a field is null, we dont need the quotes around it.
>>
>> Example:
>> "A",,"B","C"
>> Thanks
>> Anil
>>
>> On Thu, Jul 11, 2019, 1:51 PM Swetha Ramaiah <swetha.ramaiah@gmail.com>
>> wrote:
>>
>>> If you are using Spark 2.4.0, I think you can try something like this:
>>>
>>> .option("quote", "\u0000")
>>> .option("emptyValue", “”)
>>>
>>> .option("nullValue", null)
>>>
>>> Regards
>>> Swetha
>>>
>>>
>>>
>>> On Jul 11, 2019, at 1:45 PM, Anil Kulkarni <anil77k@gmail.com> wrote:
>>>
>>> Hi Spark users,
>>>
>>> My question is :
>>> I am writing a Dataframe to csv.
>>> Option i am using as
>>> .option("quoteAll","true").
>>>
>>> This is quoting even null values and making them appear as an empty
>>> string.
>>>
>>> How do i make sure that quotes are enabled only for non null values?
>>>
>>> --
>>> Cheers,
>>> Anil Kulkarni
>>> about.me/anilkulkarni
>>> [image: Anil Kulkarni on about.me]
>>>  http://anilkulkarni.com/ <http://about.me/anilkulkarni>
>>>
>>>
>>>
>> --
Regards,
Swetha

Mime
View raw message