spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Nerothin <jasonnerot...@gmail.com>
Subject Re: Handle Null Columns in Spark Structured Streaming Kafka
Date Mon, 29 Apr 2019 23:27:38 GMT
Spark SQL has had an na.fill function on it since at least 2.1. Would that
work for you?

https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html

On Mon, Apr 29, 2019 at 4:57 PM Shixiong(Ryan) Zhu <shixiong@databricks.com>
wrote:

> Hey Snehasish,
>
> Do you have a reproducer for this issue?
>
> Best Regards,
> Ryan
>
>
> On Wed, Apr 24, 2019 at 7:24 AM SNEHASISH DUTTA <info.snehasish@gmail.com>
> wrote:
>
>> Hi,
>>
>> While writing to kafka using spark structured streaming , if all the
>> values in certain column are Null it gets dropped
>> Is there any way to override this , other than using na.fill functions
>>
>> Regards,
>> Snehasish
>>
>

-- 
Thanks,
Jason

Mime
View raw message