sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Kemper (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
Date Mon, 17 Oct 2016 16:56:58 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582749#comment-15582749
] 

Markus Kemper commented on SQOOP-3025:
--------------------------------------

Hey Ruslan,

Are you able to provide the following:
1. On the host executing the Sqoop CLI command what are your security settings for [hive conf]
and [hadoop conf]
2. Our [sqoop test] appears to be working without error

[hive conf]
$ cat /etc/hive/conf/*.xml | egrep -B 1 -A 1 -i "security.authentication|hadoop.security.authorization<"
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>

[hadoop conf]
$ cat /etc/hadoop/conf/*.xml | egrep -B 1 -A 1 -i "security.authentication|hadoop.security.authorization<"
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>

[sqoop test]
$ sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from
t1"

Output:
-----------------------------------------------
| C1                   | C2                   | 
-----------------------------------------------
| 1                    | one                  | 
-----------------------------------------------

$ sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir
/user/xxx/t1_text --delete-target-dir --num-mappers 1 --as-textfile --hive-import --hive-database
default --hive-table t1_text --verbose

$ hive -e "use default; create table t1_parquet stored as parquet as select * from t1_text;"
$ sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --num-mappers
2 --hcatalog-database default --hcatalog-table t1_parquet --direct

Output:
16/10/17 09:01:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.4-SNAPSHOT
16/10/17 09:01:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
16/10/17 09:01:16 INFO manager.SqlManager: Using default fetchSize of 1000
16/10/17 09:01:18 INFO oracle.OraOopOracleQueries: Current schema is: SQOOP
16/10/17 09:01:18 INFO oracle.OraOopManagerFactory: 
**************************************************
*** Using Data Connector for Oracle and Hadoop ***
**************************************************
<SNIP>
16/10/17 09:01:24 INFO client.RMProxy: Connecting to ResourceManager at xxx.cloudera.com/xxx.xx.xx.xxx:8032
16/10/17 09:01:25 INFO hdfs.DFSClient: Created token for admin: HDFS_DELEGATION_TOKEN owner=xxx@XXX.CLOUDERA.COM,
renewer=yarn, realUser=, issueDate=1476720085263, maxDate=1477324885263, sequenceNumber=8,
masterKeyId=6 on ha-hdfs:ns1
16/10/17 09:01:25 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: HDFS_DELEGATION_TOKEN,
Service: ha-hdfs:ns1, Ident: (token for admin: HDFS_DELEGATION_TOKEN owner=xxx@XXX.CLOUDERA.COM,
renewer=yarn, realUser=, issueDate=1476720085263, maxDate=1477324885263, sequenceNumber=8,
masterKeyId=6)
<SNIP>
16/10/17 09:02:04 INFO mapreduce.Job:  map 0% reduce 0%
16/10/17 09:02:22 INFO mapreduce.Job:  map 100% reduce 0%
16/10/17 09:02:22 INFO mapreduce.Job: Job job_1476708184468_0005 completed successfully
<SNIP>
16/10/17 09:02:22 INFO mapreduce.ExportJobBase: Transferred 8.1641 KB in 57.6968 seconds (144.8954
bytes/sec)
16/10/17 09:02:22 INFO mapreduce.ExportJobBase: Exported 1 records.

$ sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from
t1"

Output:
-----------------------------------------------
| C1                   | C2                   | 
-----------------------------------------------
| 1                    | one                  | 
| 1                    | one                  | 
-----------------------------------------------

Thanks, Markus

> Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
> --------------------------------------------------------------------------------
>
>                 Key: SQOOP-3025
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3025
>             Project: Sqoop
>          Issue Type: Bug
>          Components: build, codegen, connectors/oracle, hive-integration, metastore
>    Affects Versions: 1.4.6
>         Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c
>            Reporter: Ruslan Dautkhanov
>         Attachments: scoop_output.log, sqoop_OracleDirect_mapper_nullPointer_exception.txt
>
>
> Getting following exception in one of the mappers stack when try to export to Oracle
from Hive Parquet 
> {noformat}
> 2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert:
The following error occurred during configurePreparedStatement()
> java.lang.NullPointerException
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
> 	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> 2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert:
Time spent performing an "exchange subpartition with table": 0.046365 sec.
> 2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert:
Dropping temporary mapper table "ORAOOP_20161014_231320_3"
> 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception
during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
> java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
> 	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLException: java.lang.NullPointerException
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
> 	at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
> 	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
> {noformat}
> Will attach full log file as an attachment.
> A minimal sqoop export parameters that reproduce this problem:
> {quote}
> sqoop export -Dmapred.map.max.attempts=1 --connect "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch"
--username DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg --hcatalog-database
disc_dv --hcatalog-table amf_trans_dv_09142016 --num-mappers 2 --columns "individ,part_code"
--direct --verbose 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message