spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 杨浩 <yangha...@gmail.com>
Subject double quota is automaticly added when sinking as csv
Date Tue, 21 May 2019 12:03:43 GMT
We use struct streaming 2.2, when sinking as csv, a json str will automatic
add "" for it, like an element is
>
> {"hello": "world"}

result data in fs will be

> "{\"hello\": \"world\"}"


How to avoid the "",we only want

> {"hello": "world"}

code like

>     resultDS.
>       writeStream.
>       outputMode(OutputMode.Append()).
>       trigger(Trigger.ProcessingTime(TriggerInterval, TimeUnit.SECONDS)).
>       format("csv").
>       option("sep", "\t").
>       option("path", DIR).
>       option("checkpointLocation", CheckPointDir).
>       option("compression", "gzip").
>       partitionBy("event_day", "event_hour").
>       start.
>       awaitTermination()
>

Mime
View raw message