spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <>
Subject Re: handling of empty partitions
Date Mon, 09 Jan 2017 04:30:17 GMT

Hi Georg,

Can you describe your question more clear?

Actually, the example codes you posted in stackoverflow doesn't crash as you
said in the post.

geoHeil wrote
> I am working on building a custom ML pipeline-model / estimator to impute
> missing values, e.g. I want to fill with last good known value.
> Using a window function is slow / will put the data into a single
> partition.
> I built some sample code to use the RDD API however, it some None / null
> problems with empty partitions.
> How should this be implemented properly to handle such empty partitions?
> Kind regards,
> Georg

Liang-Chi Hsieh | @viirya 
Spark Technology Center 
View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe e-mail:

View raw message