From user-return-10206-apmail-drill-user-archive=drill.apache.org@drill.apache.org Tue Mar 12 19:17:53 2019 Return-Path: X-Original-To: apmail-drill-user-archive@www.apache.org Delivered-To: apmail-drill-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CD8FB1926C for ; Tue, 12 Mar 2019 19:17:53 +0000 (UTC) Received: (qmail 53745 invoked by uid 500); 12 Mar 2019 19:17:53 -0000 Delivered-To: apmail-drill-user-archive@drill.apache.org Received: (qmail 53647 invoked by uid 500); 12 Mar 2019 19:17:53 -0000 Mailing-List: contact user-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@drill.apache.org Delivered-To: mailing list user@drill.apache.org Received: (qmail 53636 invoked by uid 99); 12 Mar 2019 19:17:53 -0000 Received: from mail-relay.apache.org (HELO mailrelay2-lw-us.apache.org) (207.244.88.137) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Mar 2019 19:17:53 +0000 Received: from BBenZvi-E754-MBP13.local (unknown [66.194.15.49]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id 7D15D3593 for ; Tue, 12 Mar 2019 19:17:52 +0000 (UTC) Reply-To: boaz@apache.org Subject: Re: RESOURCE ERROR: External Sort encountered an error while spilling to disk To: user@drill.apache.org References: From: Boaz Ben-Zvi Organization: Apache Drill Message-ID: <1c460cb8-3cf3-1d7a-02f5-8d4d451eb65a@apache.org> Date: Tue, 12 Mar 2019 12:17:51 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US  Hi Giovanni,     The error given by the External-Sort indicates a problem while spilling the excess memory into disk. When you enlarged the memory (from the default 2GB) to 8GB the LAG query may succeeded without spilling, hence circumvented the issue. Yes, you can keep enlarging the memory and run w/o spilling, but better check and fix the root issue. How is your spilling configured - check the filesystem - "drill.exec.spill.fs" and the directories - "drill.exec.spill.directories" The default is the local filesystem, and into /tmp .  It is possible that very little disk space is available for /tmp .      Thanks,          -- Boaz On 3/12/19 6:03 AM, Giovanni Conte wrote: > Hello, > I am doing a LAG and a SUM query over a PCAP dataset of 300Mb. > I get this error: > RESOURCE ERROR: External Sort encountered an error while spilling to disk > > Then I changetd > planner.memory.max_query_memory_per_node ---> 8589934592 > and with this I am able to perform the LAG but not the SUM. > > I have no problem of memory since I am working on a server with 72 cores > and 256 GB of RAM. > Which other parameter should I change to avoid the RESOURCE ERROR? > Can I go over 8589934592? > Thank you very much, > > Giovanni >