spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Peletz <alexand...@slalom.com>
Subject pyspark.sql.functions.last not working as expected
Date Wed, 17 Aug 2016 15:56:32 GMT
Hi,

I am using Spark 2.0 and I am getting unexpected results using the last() method. Has anyone
else experienced this? I get the sense that last() is working correctly within a given data
partition but not across the entire RDD. First() seems to work as expected so I can work around
this by having a window that is in reverse order and use first() instead of last() but it
would be great if last() actually worked.


Thanks,
Alexander


Alexander Peletz
Consultant

slalom

Fortune 100 Best Companies to Work For 2016
Glassdoor Best Places to Work 2016
Consulting Magazine Best Firms to Work For 2015

316 Stuart Street, Suite 300
Boston, MA 02116
706.614.5033 cell | 617.316.5400 office
alexanderp@slalom.com<mailto:alexanderp@slalom.com>


Mime
View raw message