From user-return-10151-apmail-drill-user-archive=drill.apache.org@drill.apache.org Wed Feb 13 20:23:53 2019 Return-Path: X-Original-To: apmail-drill-user-archive@www.apache.org Delivered-To: apmail-drill-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0EC3810602 for ; Wed, 13 Feb 2019 20:23:53 +0000 (UTC) Received: (qmail 19948 invoked by uid 500); 13 Feb 2019 06:57:13 -0000 Delivered-To: apmail-drill-user-archive@drill.apache.org Received: (qmail 19869 invoked by uid 500); 13 Feb 2019 06:57:13 -0000 Mailing-List: contact user-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@drill.apache.org Delivered-To: mailing list user@drill.apache.org Received: (qmail 19856 invoked by uid 99); 13 Feb 2019 06:57:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Feb 2019 06:57:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 58A1AC0584 for ; Wed, 13 Feb 2019 06:57:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.798 X-Spam-Level: * X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id uZeLvJmlt9Y8 for ; Wed, 13 Feb 2019 06:57:07 +0000 (UTC) Received: from sonic305-20.consmr.mail.ne1.yahoo.com (sonic305-20.consmr.mail.ne1.yahoo.com [66.163.185.146]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5F8075FBD6 for ; Wed, 13 Feb 2019 06:57:07 +0000 (UTC) X-YMail-OSG: YVBWgQcVM1mzUvxY_mcG6hhatoB.K6TRl_FW93jQibfxJaaYhDvvGQMv_h_u2Oh VQQeZo8q.6EkqjPeVK7sn6JtPV7UOFg9vYkomlCJI8GHwNgIruV4b9bEQlhdywKGdgNYROmV3VCo C7bcj9MQcTy.MSaI_ZFSI5esKa2o5iFAAb.MkMrPIoQhFGATG0yaccf2MCU6hZD4OaDjrM2GhTFB sFwLxoeov6VnaWlz1RXoxJQJVXZAuNuiAcEn_9YIFJhArY0_MOmX3kmB7hhcPjGPzR.bY9av9APq w2R69S4HkfmzVDW3RjQYq5KAGk1hA3MmzowrnPRkBeXb_layP1i7CnlNgxh0wsPylwKOWOlf_QJI .zWYAlb5KKItQE3FgJTqtDgoXRwiQwBxSGYowoiShIz6sZr1C5MDPFeGwHXfbCeLX2fHmp6HZ_3y 1ygK6mtw9rHv2sz_h0Ro8Ao.XuRF03m96s4Bdh9Y7vgbGjcU5HvueU.n.9CRXGkI6RGoaHtX1Hna 4kEZ42UvJN3h2l5JncLL444Le9dgEDyT9b9AfRQswYjXnJ9Q3GgNZF6.8rMJriRBGoJ8N50XZgpv kx.ZEWN1F95YxBCad1me2WQb9niItPetfhe2lDibiCVYebasELSP4BsXYw6C2pA0wbL4DPkQ_0Pm W5CI.sW8sToZOvUCWlbH2.IjFnRlRNTIcFHyXqkf2MhktueSHtHglwUm2y8XmosCw4iUTq.bV2Ch s_sn3oJBhdTXm5ZDqJIHWHPGQxAb50mZAa.hZKtoqPrCkheZxkE4E2Fkr87kKuP48A6Tu9ReBpEu 65UJFVY.GuvYf4Ug9k9BD0JjzqiE_RQJv248Jz0E2KNS6XHD_TpLYARL2MyaTcm9EUg.TkqV8bZM 9Vp3FzZhUe36DrF.YsabQ7fpCW8siculhR1W40wsUUJC8kDs3aH_Q.MLSSAr1vaOOZt84jgcw1Tx weysoKHCs6ohkisE.wyNoYY0H1Mc7CUjI7T1EvSk2iLIUyCY0l6TZMTbG.MzCvUmWDEZ9mIFVHrI W6GEzOZpkKAyx7UvLCZNBupWmbKOi94coR_gOfyuj7Y9rTmDQFy86 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.ne1.yahoo.com with HTTP; Wed, 13 Feb 2019 06:57:00 +0000 Date: Wed, 13 Feb 2019 06:56:57 +0000 (UTC) From: Krishnanand Khambadkone To: user Message-ID: <448365197.114975.1550041017527@mail.yahoo.com> In-Reply-To: References: <557590963.2609132.1550011022190.ref@mail.yahoo.com> <557590963.2609132.1550011022190@mail.yahoo.com> <1550012698658.1742126499@boxbe> <919079170.2594310.1550018310650@mail.yahoo.com> Subject: Re: HDFS storage prefix returning Error: VALIDATION ERROR: null MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_114974_1544790999.1550041017526" X-Mailer: WebService/1.1.13041 YMailNorrin Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36 ------=_Part_114974_1544790999.1550041017526 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The command show files in dfs.tmp does return the right output. However when I try to run a simple hdfs query=C2=A0 select s.application_id=C2=A0=C2=A0from=C2=A0hdfs.`/user/hive/spark_data/dt= =3D2019-01-25/part-00004-ae91cbe2-5410-4bec-ad68-10a053fb2b68.json`=C2=A0 it returns,=C2=A0=C2=A0 Error: VALIDATION ERROR: Schema [[hdfs]] is not valid with respect to eithe= r root schema or current default schema. On Tuesday, February 12, 2019, 5:10:57 PM PST, Abhishek Girish wrote: =20 =20 Can you please share the full error message (please see [1]) Also, can you please see if this works: show files in dfs.tmp; This is to check if the DFS plugin is successfully initialized and Drill can see the files on HDFS. And if that works, check if simpler queries on the data works: select * from hdfs.`` [1] https://drill.apache.org/docs/troubleshooting/#enable-verbose-errors On Tue, Feb 12, 2019 at 4:38 PM Krishnanand Khambadkone wrote: >=C2=A0 Here is the hdfs storage definition and query I am using.=C2=A0 Sam= e query > runs fine if run off local filesystem with dfs storage prefix.=C2=A0 All = I am > doing is swapping dfs for hdfs. > > { > >=C2=A0 "type": "file", > >=C2=A0 "connection": "hdfs://host18-namenode:8020/", > >=C2=A0 "config": null, > >=C2=A0 "workspaces": { > >=C2=A0 =C2=A0 "tmp": { > >=C2=A0 =C2=A0 =C2=A0 "location": "/tmp", > >=C2=A0 =C2=A0 =C2=A0 "writable": true, > >=C2=A0 =C2=A0 =C2=A0 "defaultInputFormat": null, > >=C2=A0 =C2=A0 =C2=A0 "allowAccessOutsideWorkspace": false > >=C2=A0 =C2=A0 }, > >=C2=A0 =C2=A0 "root": { > >=C2=A0 =C2=A0 =C2=A0 "location": "/", > >=C2=A0 =C2=A0 =C2=A0 "writable": false, > >=C2=A0 =C2=A0 =C2=A0 "defaultInputFormat": null, > >=C2=A0 =C2=A0 =C2=A0 "allowAccessOutsideWorkspace": false > >=C2=A0 =C2=A0 } > >=C2=A0 }, > >=C2=A0 "formats": null, > >=C2=A0 "enabled": true > > } > > > > > select s.application_id, > get_spark_attrs(s.spark_event,'spark.executor.memory') as spark_attribute= s >=C2=A0 from > hdfs.`/user/hive/spark_data/dt=3D2019-01-25/part-00004-ae91cbe2-5410-4bec= -ad68-10a053fb2b68.json` > s where (REGEXP_REPLACE(REGEXP_REPLACE(substr(s.spark_event,11), > '[^0-9A-Za-z]"', ''),'(".*)','') =3D 'SparkListenerEnvironmentUpdate' or > REGEXP_REPLACE(REGEXP_REPLACE(substr(s.spark_event,11), '[^0-9A-Za-z]"', > ''),'(".*)','') =3D 'SparkListenerApplicationStart' or > REGEXP_REPLACE(REGEXP_REPLACE(substr(s.spark_event,11), '[^0-9A-Za-z]"', > ''),'(".*)','') =3D 'SparkListenerApplicationEnd') group by application_i= d, > spark_attributes=C2=A0 order by application_id; > > > >=C2=A0 =C2=A0 On Tuesday, February 12, 2019, 3:04:40 PM PST, Abhishek Giri= sh < > agirish@apache.org> wrote: > >=C2=A0 This message is eligible for Automatic Cleanup! (agirish@apache.org= ) Add > cleanup rule | More info >=C2=A0 Hey Krishnanand, > > As mentioned by other folks in earlier threads, can you make sure to > include ALL RELEVANT details in your emails? That includes the query, > storage plugin configuration, data format, sample data / description of t= he > data, the full log for the query failure? It's necessary if one needs to = be > able to understand the issue or offer help. > > Regards, > Abhishek > > On Tue, Feb 12, 2019 at 2:37 PM Krishnanand Khambadkone > wrote: > > > I have defined a hdfs storage type with all the required properties. > > However, when I try to use that in the query it returns > > Error: VALIDATION ERROR: null > > > =20 ------=_Part_114974_1544790999.1550041017526--