Spark SQL has had an na.fill function on it since at least 2.1. Would that work for you?

https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html

On Mon, Apr 29, 2019 at 4:57 PM Shixiong(Ryan) Zhu <shixiong@databricks.com> wrote:
Hey Snehasish,

Do you have a reproducer for this issue?

Best Regards,

Ryan


On Wed, Apr 24, 2019 at 7:24 AM SNEHASISH DUTTA <info.snehasish@gmail.com> wrote:
Hi,

While writing to kafka using spark structured streaming , if all the values in certain column are Null it gets dropped
Is there any way to override this , other than using na.fill functions

Regards,
Snehasish


--
Thanks,
Jason