Caused by: java.lang.IllegalArgumentException: Cannot allocate a page with more than 17179869176 bytes
Same error for:
sql("SELECT " + field + " FROM MY_TABLE).sort(field)
sql("SELECT " + field + " FROM MY_TABLE).orderBy(field)
I'm running this on a machine with more than 200GB of RAM, running in local mode with spark.driver.memory set to 64g.
I do not know why it cannot allocate a big enough page, and why is it trying to allocate such a big page in the first place?
I hope someone with more knowledge of Spark can shed some light on this. Thank you!