spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tiandiwoxin <>
Subject TakeOrderedAndProject operator may causes an OOM
Date Thu, 04 Feb 2016 09:19:52 GMT

Currently the TakeOrderedAndProject operator in spark sql uses RDD’s
takeOrdered method. When we pass a large limit to operator, however, it will
return partitionNum*limit number of records to the driver which may cause an

Are there any plans to deal with the problem in the community? 



View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message