drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Updike, Clark" <Clark.Upd...@jhuapl.edu>
Subject Re: RE: HDFS file is listable but not queryable (object not found)
Date Thu, 23 Jul 2020 18:26:02 GMT
I should mention that this is a kerberized HDFS cluster.  I'm still not sure why the SHOW FILES
would work but the query would not--but it could be behind the issue somehow.

´╗┐On 7/23/20, 2:18 PM, "Updike, Clark" <Clark.Updike@jhuapl.edu> wrote:
 
    No change unfortunately:
    
    apache drill> select * from hdfs.`root`.`/tmp/employee.json`;
    Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 18: Object '/tmp/employee.json'
not found within 'hdfs.root'
    
    On 7/23/20, 2:11 PM, "Paul Rogers" <par0328@gmail.com> wrote:
    
        Hi Clark,
        
        Try using `hdfs`.`root` rather than `hdfs.root`. Calcite wants to walk down
        `hdfs` then `root`. There is no workspace called `hdfs.root`.
        
        Thanks,
        
        - Paul
        
        On Thu, Jul 23, 2020 at 8:58 AM Updike, Clark <Clark.Updike@jhuapl.edu>
        wrote:
        
        > Oops, sorry.  No luck there either unfortunately:
        >
        > apache drill> SELECT * FROM hdfs.`/tmp/employee.json`;
        > Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 18:
        > Object '/tmp/employee.json' not found within 'hdfs'
        >
        >
        > On 7/23/20, 11:52 AM, "Charles Givre" <cgivre@gmail.com> wrote:
        >
        >     Oh.. I meant:
        >
        >     SELECT *
        >     FROM hdfs.`/tmp/employee.json`
        >
        >     > On Jul 23, 2020, at 11:41 AM, Updike, Clark <Clark.Updike@jhuapl.edu>
        > wrote:
        >     >
        >     > No change unfortunately...
        >     >
        >     > $ hdfs dfs -ls hdfs://nn01:8020/tmp/employee.json
        >     > -rw-r--r--   2 me supergroup     474630 2020-07-23 10:53
        > hdfs://nn01:8020/tmp/employee.json
        >     >
        >     > apache drill> select * from
        > hdfs.root.`hdfs://nn01:8020/tmp/employee.json`;
        >     > Error: VALIDATION ERROR: From line 1, column 15 to line 1, column
        > 18: Object 'hdfs://nn01:8020/tmp/employee.json' not found within 'hdfs.root'
        >     >
        >     >
        >     > On 7/23/20, 11:30 AM, "Charles Givre" <cgivre@gmail.com> wrote:
        >     >
        >     >    Hi Clark,
        >     >    That's strange.  My initial thought is that this could be a
        > permission issue.  However, it might also be that Drill isn't finding the
        > file for some reason.
        >     >
        >     >    Could you try:
        >     >
        >     >    SELECT *
        >     >    FROM hdfs.`<full hdfs path to file>`
        >     >
        >     >    Best,
        >     >    --- C
        >     >
        >     >
        >     >> On Jul 23, 2020, at 11:23 AM, Updike, Clark <
        > Clark.Updike@jhuapl.edu> wrote:
        >     >>
        >     >> This is in 1.17.  I can use SHOW FILES to list the file I'm
        > targeting, but I cannot query it:
        >     >>
        >     >> apache drill> show files in hdfs.root.`/tmp/employee.json`;
        >     >>
        > +---------------+-------------+--------+--------+----------+------------+-------------+-------------------------+-------------------------+
        >     >> |     name      | isDirectory | isFile | length |  owner   |
        >  group    | permissions |       accessTime        |    modificationTime
        >  |
        >     >>
        > +---------------+-------------+--------+--------+----------+------------+-------------+-------------------------+-------------------------+
        >     >> | employee.json | false       | true   | 474630 | me       |
        > supergroup | rw-r--r--   | 2020-07-23 10:53:15.055 | 2020-07-23
        > 10:53:15.387 |
        >     >>
        > +---------------+-------------+--------+--------+----------+------------+-------------+-------------------------+-------------------------+
        >     >> 1 row selected (3.039 seconds)
        >     >>
        >     >>
        >     >> apache drill> select * from hdfs.root.`/tmp/employee.json`;
        >     >> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column
        > 18: Object '/tmp/employee.json' not found within 'hdfs.root'
        >     >> [Error Id: 3b833622-4fac-4ecc-becd-118291cd8560 ] (state=,code=0)
        >     >>
        >     >> The storage plugin uses the standard json config:
        >     >>
        >     >>   "json": {
        >     >>     "type": "json",
        >     >>     "extensions": [
        >     >>       "json"
        >     >>     ]
        >     >>   },
        >     >>
        >     >> I can't see any problems on the HDFS side.  Full stack trace is
        > below.
        >     >>
        >     >> Any ideas what could be causing this behavior?
        >     >>
        >     >> Thanks, Clark
        >     >>
        >     >>
        >     >>
        >     >> FULL STACKTRACE:
        >     >>
        >     >> apache drill> select * from hdfs.root.`/tmp/employee.json`;
        >     >> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column
        > 18: Object '/tmp/employee.json' not found within 'hdfs.root'
        >     >>
        >     >>
        >     >> [Error Id: 69c8ffc0-4933-4008-a786-85ad623578ea ]
        >     >>
        >     >> (org.apache.calcite.runtime.CalciteContextException) From line 1,
        > column 15 to line 1, column 18: Object '/tmp/employee.json' not found
        > within 'hdfs.root'
        >     >>   sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
        >     >>   sun.reflect.NativeConstructorAccessorImpl.newInstance():62
        >     >>   sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
        >     >>   java.lang.reflect.Constructor.newInstance():423
        >     >>   org.apache.calcite.runtime.Resources$ExInstWithCause.ex():463
        >     >>   org.apache.calcite.sql.SqlUtil.newContextException():824
        >     >>   org.apache.calcite.sql.SqlUtil.newContextException():809
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():4805
        >     >>
        >  org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl():127
        >     >>
        >  org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl():177
        >     >>   org.apache.calcite.sql.validate.AbstractNamespace.validate():84
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():995
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():955
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3109
        >     >>
        >  org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom():298
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3091
        >     >>
        >  org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom():298
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3363
        >     >>   org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
        >     >>   org.apache.calcite.sql.validate.AbstractNamespace.validate():84
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():995
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():955
        >     >>   org.apache.calcite.sql.SqlSelect.validate():216
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():930
        >     >>   org.apache.calcite.sql.validate.SqlValidatorImpl.validate():637
        >     >>   org.apache.drill.exec.planner.sql.SqlConverter.validate():218
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():665
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():199
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
        >     >>
        >  org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():282
        >     >>
        >  org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():162
        >     >>   org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():127
        >     >>   org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():92
        >     >>   org.apache.drill.exec.work.foreman.Foreman.runSQL():590
        >     >>   org.apache.drill.exec.work.foreman.Foreman.run():275
        >     >>   java.util.concurrent.ThreadPoolExecutor.runWorker():1142
        >     >>   java.util.concurrent.ThreadPoolExecutor$Worker.run():617
        >     >>   java.lang.Thread.run():745
        >     >> Caused By (org.apache.calcite.sql.validate.SqlValidatorException)
        > Object '/tmp/employee.json' not found within 'hdfs.root'
        >     >>   sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
        >     >>   sun.reflect.NativeConstructorAccessorImpl.newInstance():62
        >     >>   sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
        >     >>   java.lang.reflect.Constructor.newInstance():423
        >     >>   org.apache.calcite.runtime.Resources$ExInstWithCause.ex():463
        >     >>   org.apache.calcite.runtime.Resources$ExInst.ex():572
        >     >>   org.apache.calcite.sql.SqlUtil.newContextException():824
        >     >>   org.apache.calcite.sql.SqlUtil.newContextException():809
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():4805
        >     >>
        >  org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl():127
        >     >>
        >  org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl():177
        >     >>   org.apache.calcite.sql.validate.AbstractNamespace.validate():84
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():995
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():955
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3109
        >     >>
        >  org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom():298
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3091
        >     >>
        >  org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom():298
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3363
        >     >>   org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
        >     >>   org.apache.calcite.sql.validate.AbstractNamespace.validate():84
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():995
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():955
        >     >>   org.apache.calcite.sql.SqlSelect.validate():216
        >     >>
        >  org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():930
        >     >>   org.apache.calcite.sql.validate.SqlValidatorImpl.validate():637
        >     >>   org.apache.drill.exec.planner.sql.SqlConverter.validate():218
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():665
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():199
        >     >>
        >  org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():172
        >     >>
        >  org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():282
        >     >>
        >  org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():162
        >     >>   org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():127
        >     >>   org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():92
        >     >>   org.apache.drill.exec.work.foreman.Foreman.runSQL():590
        >     >>   org.apache.drill.exec.work.foreman.Foreman.run():275
        >     >>   java.util.concurrent.ThreadPoolExecutor.runWorker():1142
        >     >>   java.util.concurrent.ThreadPoolExecutor$Worker.run():617
        >     >>   java.lang.Thread.run():745 (state=,code=0)
        >     >
        >     >
        >     >
        >
        >
        >
        >
        
    
    

Mime
View raw message