spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <vii...@gmail.com>
Subject Re: handling of empty partitions
Date Mon, 09 Jan 2017 04:30:17 GMT

Hi Georg,

Can you describe your question more clear?

Actually, the example codes you posted in stackoverflow doesn't crash as you
said in the post.


geoHeil wrote
> I am working on building a custom ML pipeline-model / estimator to impute
> missing values, e.g. I want to fill with last good known value.
> Using a window function is slow / will put the data into a single
> partition.
> I built some sample code to use the RDD API however, it some None / null
> problems with empty partitions.
> 
> How should this be implemented properly to handle such empty partitions?
> http://stackoverflow.com/questions/41474175/spark-mappartitionswithindex-handling-empty-partitions
> 
> Kind regards,
> Georg





-----
Liang-Chi Hsieh | @viirya 
Spark Technology Center 
http://www.spark.tc/ 
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/handling-of-empty-partitions-tp20496p20515.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message