drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz Ben-Zvi <b...@apache.org>
Subject Re: RESOURCE ERROR: External Sort encountered an error while spilling to disk
Date Tue, 12 Mar 2019 19:17:51 GMT
  Hi Giovanni,

     The error given by the External-Sort indicates a problem while 
spilling the excess memory into disk.

When you enlarged the memory (from the default 2GB) to 8GB the LAG query 
may succeeded without spilling, hence circumvented the issue.

Yes, you can keep enlarging the memory and run w/o spilling, but better 
check and fix the root issue.

How is your spilling configured - check the filesystem - 
"drill.exec.spill.fs" and the directories - "drill.exec.spill.directories"

The default is the local filesystem, and into /tmp .  It is possible 
that very little disk space is available for /tmp .

      Thanks,

          -- Boaz

On 3/12/19 6:03 AM, Giovanni Conte wrote:
> Hello,
> I am doing a LAG and a SUM query over a PCAP dataset of 300Mb.
> I get this error:
> RESOURCE ERROR: External Sort encountered an error while spilling to disk
>
> Then I changetd
> planner.memory.max_query_memory_per_node ---> 8589934592
> and with this I am able to perform the LAG but not the SUM.
>
> I have no problem of memory since I am working on a server with 72 cores
> and 256 GB of RAM.
> Which other parameter should I change to avoid the RESOURCE ERROR?
> Can I go over 8589934592?
> Thank you very much,
>
> Giovanni
>

Mime
View raw message