spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Nerothin <jasonnerot...@gmail.com>
Subject Re: Handle Null Columns in Spark Structured Streaming Kafka
Date Mon, 29 Apr 2019 23:28:43 GMT
See also here:
https://stackoverflow.com/questions/44671597/how-to-replace-null-values-with-a-specific-value-in-dataframe-using-spark-in-jav

On Mon, Apr 29, 2019 at 5:27 PM Jason Nerothin <jasonnerothin@gmail.com>
wrote:

> Spark SQL has had an na.fill function on it since at least 2.1. Would that
> work for you?
>
>
> https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html
>
> On Mon, Apr 29, 2019 at 4:57 PM Shixiong(Ryan) Zhu <
> shixiong@databricks.com> wrote:
>
>> Hey Snehasish,
>>
>> Do you have a reproducer for this issue?
>>
>> Best Regards,
>> Ryan
>>
>>
>> On Wed, Apr 24, 2019 at 7:24 AM SNEHASISH DUTTA <info.snehasish@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> While writing to kafka using spark structured streaming , if all the
>>> values in certain column are Null it gets dropped
>>> Is there any way to override this , other than using na.fill functions
>>>
>>> Regards,
>>> Snehasish
>>>
>>
>
> --
> Thanks,
> Jason
>


-- 
Thanks,
Jason

Mime
View raw message