spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <akshay.bhardwaj1...@gmail.com>
Subject Re: double quota is automaticly added when sinking as csv
Date Tue, 21 May 2019 12:18:49 GMT
Hi,

Add writeStream.option("quoteMode", "NONE")

By default Spark dataset api assumes that all the values MUST BE enclosed
in quote character (def: ") while writing to CSV files.

Akshay Bhardwaj
+91-97111-33849


On Tue, May 21, 2019 at 5:34 PM 杨浩 <yanghaogn@gmail.com> wrote:

> We use struct streaming 2.2, when sinking as csv, a json str will automatic
> add "" for it, like an element is
>>
>> {"hello": "world"}
>
> result data in fs will be
>
>> "{\"hello\": \"world\"}"
>
>
> How to avoid the "",we only want
>
>> {"hello": "world"}
>
> code like
>
>>     resultDS.
>>       writeStream.
>>       outputMode(OutputMode.Append()).
>>       trigger(Trigger.ProcessingTime(TriggerInterval, TimeUnit.SECONDS)).
>>       format("csv").
>>       option("sep", "\t").
>>       option("path", DIR).
>>       option("checkpointLocation", CheckPointDir).
>>       option("compression", "gzip").
>>       partitionBy("event_day", "event_hour").
>>       start.
>>       awaitTermination()
>>
>
>

Mime
View raw message