spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Helper methods for PySpark discussion
Date Fri, 26 Oct 2018 16:15:35 GMT
Coming out of it was agreed the
helper methods in question made sense but there was some desire for a plan
as to which helper methods we should use.

I'd like to purpose a light weight solution to start with for helper
methods that match either Pandas or general Python collection helper
1) If the helper method doesn't collect the DataFrame back or force
evaluation to the driver then we should add it without discussion
2) If the method forces evaluation this matches most obvious way that would
implemented then we should add it with a note in the docstring
3) If the method does collect the DataFrame back to the driver and that is
the most obvious way it would implemented (e.g. calling list to get back a
list would have to collect the DataFrame) then we should add it with a
warning in the docstring
4) If the method collects the DataFrame but a reasonable Python developer
wouldn't expect that behaviour not implementing the helper method would be

What do folks think?
Books (Learning Spark, High Performance Spark, etc.):  <>
YouTube Live Streams:

View raw message