From user-return-5783-apmail-drill-user-archive=drill.apache.org@drill.apache.org Tue May 3 16:04:02 2016 Return-Path: X-Original-To: apmail-drill-user-archive@www.apache.org Delivered-To: apmail-drill-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ED63A1933D for ; Tue, 3 May 2016 16:04:01 +0000 (UTC) Received: (qmail 31924 invoked by uid 500); 3 May 2016 16:04:01 -0000 Delivered-To: apmail-drill-user-archive@drill.apache.org Received: (qmail 31860 invoked by uid 500); 3 May 2016 16:04:01 -0000 Mailing-List: contact user-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@drill.apache.org Delivered-To: mailing list user@drill.apache.org Received: (qmail 31840 invoked by uid 99); 3 May 2016 16:04:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2016 16:04:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A9E9CC015A for ; Tue, 3 May 2016 16:04:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 3cyAjOehI5vL for ; Tue, 3 May 2016 16:03:58 +0000 (UTC) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E7AB45FCF4 for ; Tue, 3 May 2016 16:03:57 +0000 (UTC) Received: by mail-lf0-f51.google.com with SMTP id m64so27596680lfd.1 for ; Tue, 03 May 2016 09:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=e9IcO6NXsqLTmaHx8J0/1znaRwMrJ5kG/u1d8ChuiIU=; b=bwFCBnFDxyBanCTOZ3dumnaVfMhKH5cAvkfvcPpLyTN0Q2ZYc/gd3mWGK9mFgUSSQC kfhw0GLo/3ghAnYA1wnQi5V8O3D0RXzlmGIob06v8c7ujoA1waxO2M6JJ5Y7P/FwFfrZ 8RibM210w8ZKtGtvWDHTtpiPqTV6+d8xUHzV5qzity5hWGXTDzkfdhuJLm1Te4EESVOo /RSgX4PDIwvNyrCUi5YMbDksfFc+qBXIEhEMi03tZFJ1gp7UJL9KVpZA7VP5tOqU7as9 lg7xZPIqgWT22OaqX67f7UzokIWVdoOdJmP+a5sSenFvkuhYSBpOZXTf74KLSiy9Wqyi PgIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=e9IcO6NXsqLTmaHx8J0/1znaRwMrJ5kG/u1d8ChuiIU=; b=YPAW+5hxelihnb0rpreLdoXbLYxaNio/eW4d2b4oqd5tMNfyuEiuAet5doSR5s7pY1 IEkpH+sRgM1a023ugKeEvtqymSdkQU+4hTXAJEYj94InqnxpGegSUrZ4ADYYbo9TNcxB 3L/EJr2sGKUphyc+gZDMl3DGHApXIklbiALEFXsLC8ltAo2MiH1TjUwZH+AL219Zlpsn QBesOi6+2Djz1KZ+XI2ZxqU4f8S9AxTRSvGZM9syIOPTrVvLWd4+WQDTKv4iv8/m/iHO fGpbO61V7R7ACGpKNYzG2mji7LVNskUx+d3uNU0dd+EpNgc9AkXbz7BcxPg28enWu462 13UA== X-Gm-Message-State: AOPr4FVamQJSDvNmXIEHCGsNGDou8ktUwfEz7t8XD4Y6F7DXsXQQ7FCMIby2gm0wIypOEQ== X-Received: by 10.112.52.196 with SMTP id v4mr1379160lbo.59.1462291436337; Tue, 03 May 2016 09:03:56 -0700 (PDT) Received: from mail-lf0-f46.google.com (mail-lf0-f46.google.com. [209.85.215.46]) by smtp.gmail.com with ESMTPSA id zu6sm615596lbb.16.2016.05.03.09.03.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 May 2016 09:03:54 -0700 (PDT) Received: by mail-lf0-f46.google.com with SMTP id m64so27595386lfd.1 for ; Tue, 03 May 2016 09:03:54 -0700 (PDT) X-Received: by 10.25.40.145 with SMTP id o139mr1544887lfo.151.1462291434180; Tue, 03 May 2016 09:03:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.161.12 with HTTP; Tue, 3 May 2016 09:03:14 -0700 (PDT) In-Reply-To: References: From: Abhishek Girish Date: Tue, 3 May 2016 09:03:14 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: "java.lang.OutOfMemoryError: Java heap space" error which in-turn kills drill bit of one of the node To: Drill User List Content-Type: multipart/alternative; boundary=001a11403b2c7e31080531f2419b --001a11403b2c7e31080531f2419b Content-Type: text/plain; charset=UTF-8 Can you try bumping up Drill Heap memory and restarting Drillbits? Looks related to DRILL-3678 Refer to http://drill.apache.org/docs/configuring-drill-memory/ On Tue, May 3, 2016 at 3:19 AM, Anup Tiwari wrote: > Hi All, > > Sometimes I am getting below error while creating a table in drill using a > hive table :- > > *"*java.lang.OutOfMemoryError: Java heap space*"* which in-turn kills drill > bit of one of the node where i have executed respective query. > > *Query Type :-* > > create table glv_abc as select sessionid, max(serverTime) as max_serverTime > from hive.hive_logs_daily > where log_date = '2016-05-02' > group by sessionid; > > > Kindly help me in this. > > Please find *output of drillbit.log* below :- > > 2016-05-03 15:33:15,628 [28d7890f-a7d6-b55e-3853-23f1ea828751:frag:2:12] > ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurr > ed, exiting. Information message: Unable to handle out of memory condition > in FragmentExecutor. > java.lang.OutOfMemoryError: Java heap space > at > > hive.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:755) > ~[drill-hive-exec-shaded-1.6.0.jar:1.6. > 0] > at > > hive.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:494) > ~[drill-hive-exec-shaded-1.6.0.jar:1.6.0] > at > > hive.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > ~[drill-hive-exec-shaded-1.6.0.jar:1.6 > .0] > at > > hive.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) > ~[drill-hive-exec-shaded-1.6.0.jar: > 1.6.0] > at > > hive.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) > ~[drill-hive-exec-shaded-1.6.0.jar:1.6.0] > at > > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206) > ~[drill-hive-exec-shade > d-1.6.0.jar:1.6.0] > at > > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:62) > ~[drill-hive-exec-shaded > -1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.store.hive.HiveRecordReader.next(HiveRecordReader.java:321) > ~[drill-storage-hive-core-1.6.0.jar:1.6.0] > at > org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:191) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94) > ~[drill-java-exec-1.6.0.jar:1 > .6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.test.generated.HashAggregatorGen731.doWork(HashAggTemplate.java:314) > ~[na:na] > at > > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:133) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > ~[drill-java-exec-1.6.0.jar:1.6.0] > 2016-05-03 15:33:16,648 [Drillbit-ShutdownHook#0] INFO > o.apache.drill.exec.server.Drillbit - Received shutdown request. > 2016-05-03 15:33:16,669 [28d7890f-a7d6-b55e-3853-23f1ea828751:frag:2:16] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 28d7890f-a7d6-b55e-3853-23f1ea828751:2:16: State change requested RUNNING > --> FAILED > 2016-05-03 15:33:16,670 [28d7890f-a7d6-b55e-3853-23f1ea828751:frag:2:16] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 28d7890f-a7d6-b55e-3853-23f1ea828751:2:16: State change requested FAILED > --> FINISHED > 2016-05-03 15:33:16,675 [28d7890f-a7d6-b55e-3853-23f1ea828751:frag:2:16] > ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IOException: > Filesystem closed > > Fragment 2:16 > > [Error Id: 8604418f-ac5e-4e79-b66b-cd7d779b38f7 on namenode:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IOException: Filesystem closed > > > Regards, > *Anup* > --001a11403b2c7e31080531f2419b--