hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-23873) Querying Hive JDBCStorageHandler table fails with NPE when CBO is off
Date Mon, 03 Aug 2020 06:25:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23873?focusedWorklogId=465559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-465559
]

ASF GitHub Bot logged work on HIVE-23873:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Aug/20 06:24
            Start Date: 03/Aug/20 06:24
    Worklog Time Spent: 10m 
      Work Description: shameersss1 commented on a change in pull request #1328:
URL: https://github.com/apache/hive/pull/1328#discussion_r464215422



##########
File path: jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
##########
@@ -59,9 +62,26 @@ public JdbcRecordIterator(Connection conn, PreparedStatement ps, ResultSet
rs, C
       fieldNamesProperty = Preconditions.checkNotNull(conf.get(Constants.JDBC_QUERY_FIELD_NAMES));
       fieldTypesProperty = Preconditions.checkNotNull(conf.get(Constants.JDBC_QUERY_FIELD_TYPES));
     } else {
-      fieldNamesProperty = Preconditions.checkNotNull(conf.get(serdeConstants.LIST_COLUMNS));
+      try {
+        if (conf.get(Constants.JDBC_QUERY) == null) {
+          ResultSetMetaData metadata = rs.getMetaData();
+          int numColumns = metadata.getColumnCount();
+          List<String> columnNames = new ArrayList<String>(numColumns);
+          for (int i = 0; i < numColumns; i++) {
+            columnNames.add(metadata.getColumnName(i + 1));
+          }
+          fieldNamesProperty = String.join(",",columnNames);

Review comment:
       nit: space missing:  **String.join(",", columnNames);**




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 465559)
    Time Spent: 1h 10m  (was: 1h)

> Querying Hive JDBCStorageHandler table fails with NPE when CBO is off
> ---------------------------------------------------------------------
>
>                 Key: HIVE-23873
>                 URL: https://issues.apache.org/jira/browse/HIVE-23873
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, JDBC
>    Affects Versions: 3.1.0, 3.1.1, 3.1.2
>            Reporter: Chiran Ravani
>            Assignee: Chiran Ravani
>            Priority: Critical
>              Labels: pull-request-available
>         Attachments: HIVE-23873.01.patch, HIVE-23873.02.patch, HIVE-23873.3.patch, HIVE-23873.4.patch
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Scenario is Hive table having same schema as table in Oracle, however when we query the
table with data it fails with NPE, below is the trace.
> {code}
> Caused by: java.io.IOException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:617)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:524)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473)
~[hive-service-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         ... 34 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hive.storage.jdbc.JdbcSerDe.deserialize(JdbcSerDe.java:164) ~[hive-jdbc-handler-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:598)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:524)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
~[hive-exec-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473)
~[hive-service-3.1.0.3.1.5.0-152.jar:3.1.0.3.1.5.0-152]
>         ... 34 more
> {code}
> Problem appears when column names in Oracle are in Upper case and since in Hive, table
and column names are forced to store in lowercase during creation. User runs into NPE error
while fetching data.
> While deserializing data, input consists of column names in lower case which fails to
get the value
> https://github.com/apache/hive/blob/rel/release-3.1.2/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java#L136
> {code}
> rowVal = ((ObjectWritable)value).get();
> {code}
> Log Snio:
> =============
> {code}
> 2020-07-17T16:49:09,598 INFO  [04ed42ec-91d2-4662-aee7-37e840a06036 HiveServer2-Handler-Pool:
Thread-104]: dao.GenericJdbcDatabaseAccessor (:()) - Query to execute is [select * from TESTHIVEJDBCSTORAGE]
> 2020-07-17T16:49:10,642 INFO  [04ed42ec-91d2-4662-aee7-37e840a06036 HiveServer2-Handler-Pool:
Thread-104]: jdbc.JdbcSerDe (:()) - *** ColumnKey = ID
> 2020-07-17T16:49:10,642 INFO  [04ed42ec-91d2-4662-aee7-37e840a06036 HiveServer2-Handler-Pool:
Thread-104]: jdbc.JdbcSerDe (:()) - *** Blob value = {fname=OW[class=class java.lang.String,value=Name1],
id=OW[class=class java.lang.Integer,value=1]}
> {code}
> Simple Reproducer for this case.
> =============
> 1. Create table in Oracle
> {code}
> create table TESTHIVEJDBCSTORAGE(ID INT, FNAME VARCHAR(20));
> {code}
> 2. Insert dummy data.
> {code}
> Insert into TESTHIVEJDBCSTORAGE values (1, 'Name1');
> {code}
> 3. Create JDBCStorageHandler table in Hive.
> {code}
> CREATE EXTERNAL TABLE default.TESTHIVEJDBCSTORAGE_HIVE_TBL (ID INT, FNAME VARCHAR(20))

> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' 
> TBLPROPERTIES ( 
> "hive.sql.database.type" = "ORACLE", 
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver", 
> "hive.sql.jdbc.url" = "jdbc:oracle:thin:@orachehostname/XE", 
> "hive.sql.dbcp.username" = "chiran", 
> "hive.sql.dbcp.password" = "supersecurepassword", 
> "hive.sql.table" = "TESTHIVEJDBCSTORAGE", 
> "hive.sql.dbcp.maxActive" = "1" 
> );
> {code}
> 4. Query Hive table, fails with NPE.
> {code}
> > select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL;
> INFO  : Compiling command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc):
select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testhivejdbcstorage_hive_tbl.id,
type:int, comment:null), FieldSchema(name:testhivejdbcstorage_hive_tbl.fname, type:varchar(20),
comment:null)], properties:null)
> INFO  : Completed compiling command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc);
Time taken: 9.914 seconds
> INFO  : Executing command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc):
select * from default.TESTHIVEJDBCSTORAGE_HIVE_TBL
> INFO  : Completed executing command(queryId=hive_20200717164857_cd6f5020-4a69-4a2d-9e63-9db99d0121bc);
Time taken: 0.019 seconds
> INFO  : OK
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {code}
> Assuming that there are no repercussions, can we convert the column names to lowercase
fetched from Database/Query pointing to table in JDBCStorageHandler?
> Attaching the patch for the case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message