spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <>
Subject Re: SparkSQL with large result size
Date Mon, 02 May 2016 13:21:53 GMT
Please consider decreasing block size. 


> On May 1, 2016, at 9:19 PM, Buntu Dev <> wrote:
> I got a 10g limitation on the executors and operating on parquet dataset with block size
70M with 200 blocks. I keep hitting the memory limits when doing a 'select * from t1 order
by c1 limit 1000000' (ie, 1M). It works if I limit to say 100k. What are the options to save
a large dataset without running into memory issues?
> Thanks!

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message