sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Antonio Villar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-3120) Error exporting large table to Oracle and Append Values Hint (duplicated 3106)
Date Wed, 06 Sep 2017 09:08:00 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Antonio Villar updated SQOOP-3120:
----------------------------------
    Description:     (was:    I'm using Oracle Data Connector for Oracle and Hadoop to export
data from Hive to Oracle, when i use the append_values hint i get a Exception and mappers
don't end.

This only happens when i export more than 18 columns of the original table.

The command that i'm using is:

{code}
sqoop export \
  -D oraoop.partitioned=true -D oraoop.template.table=ECI_ORA.CARTERA_XPAN_PROD_EXT_15 \
  -D oraoop.oracle.append.values.hint.usage=ON \
   -connect jdbc:oracle:thin:ECI_ORA/XXXXXX@dpexr02-scan.lacaixa.es:1522/s_dpedwr_datapool.lacaixa.es
\
   --num-mappers 4  \
   --table ECI_ORA.AAA_1GIGA \
   --columns "TIINDI,CODINDI,EMPIMP,OFIIMP,EMPEMI,TICOGE,CONCSIG,CDCREP,EMPDIV,CODDIV,CODCARTE,EMPGEST,CDSEGCLI,TIPCAR,NUESTCE,TIDADCE,EQUIPO,CDSEGBAN,CODPROD,INDVALO101,INDVALO102,INDVALO103,INDVALO104,INDVALO105,INDVALO106,INDVALO107,INDVALO108,INDVALO109,INDVALO110,INDVALO111,INDVALO112,INDVALO113,INDVALO114,INDVALO115,INDVALO116,INDVALO117,INDVALO118,INDVALO119,INDVALO201,INDVALO202,INDVALO203,INDVALO204,INDVALO205,INDVALO206,INDVALO207,INDVALO208,INDVALO209,INDVALO210,INDVALO211,INDVALO212,INDVALO213,INDVALO214,INDVALO215,INDVALO216,INDVALO217,INDVALO218,INDVALO219,NIVEL"
\
  --export-dir  /apps/eciora/hive/l04_eciora.db/cartera_xpan_prod_ext_1 \
   --input-fields-terminated-by  '\01' \
  --input-lines-terminated-by  '\n' \
   --input-null-non-string   '\\\\N' \
   --input-null-string   '\\\\N' \
   --direct 
{code}

This is the log of the mapper:

{code}
2016-12-29 11:39:36,886 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2016-12-29 11:39:36,987 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2016-12-29 11:39:36,987 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system started
2016-12-29 11:39:36,998 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2016-12-29 11:39:36,998 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job,
Service: job_1481807148415_6344, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6e171cd7)
2016-12-29 11:39:37,079 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now.
2016-12-29 11:39:37,364 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir
for child: /u12/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u11/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u10/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u09/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u08/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u07/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u06/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u05/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u04/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u03/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u02/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u01/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344
2016-12-29 11:39:37,645 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id
is deprecated. Instead, use dfs.metrics.session-id
2016-12-29 11:39:38,182 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
2016-12-29 11:39:38,528 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.sqoop.mapreduce.hcat.SqoopHCatInputSplit@62f68dff
2016-12-29 11:39:38,572 WARN [main] org.apache.sqoop.manager.oracle.OraOopUtilities: System
property java.security.egd is not set to file:///dev/urandom - Oracle connections may time
out.
2016-12-29 11:39:38,575 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
The number of batch-inserts to perform per commit has been changed from 0 to 1. This is in
response to the Oracle APPEND_VALUES hint being used.
2016-12-29 11:39:38,576 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
The number of rows per batch-insert has been changed from 0 to 5000. This is in response to
the Oracle APPEND_VALUES hint being used.
2016-12-29 11:39:39,142 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
The number of rows per batch is: 5000
2016-12-29 11:39:39,142 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
The number of batches per commit is: 1
2016-12-29 11:39:39,341 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
This record writer is connected to Oracle via the JDBC URL: 
	"oracle.jdbc.driver.T4CConnection@72a85671"
	to the Oracle instance: "dpdb1rb1"
2016-12-29 11:39:39,348 INFO [main] org.apache.sqoop.manager.oracle.OraOopOracleQueries: Session
Time Zone set to GMT
2016-12-29 11:39:39,350 INFO [main] org.apache.sqoop.manager.oracle.OracleConnectionFactory:
Initializing Oracle session with SQL :
begin 
  dbms_application_info.set_module(module_name => 'Data Connector for Oracle and Hadoop',
action_name => 'export 20161229113914CET'); 
end;
2016-12-29 11:39:39,351 INFO [main] org.apache.sqoop.manager.oracle.OracleConnectionFactory:
Initializing Oracle session with SQL : alter session force parallel dml
2016-12-29 11:39:39,351 INFO [main] org.apache.sqoop.manager.oracle.OracleConnectionFactory:
Initializing Oracle session with SQL : alter session disable parallel query
2016-12-29 11:39:39,500 INFO [main] org.apache.hive.hcatalog.mapreduce.InternalUtil: Initializing
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe with properties {name=l04_eciora.cartera_xpan_prod_ext_10000_rows,
numFiles=1, columns.types=int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),int,
serialization.format=1, columns=tiindi,codindi,empimp,ofiimp,empemi,ticoge,concsig,cdcrep,empdiv,coddiv,codcarte,empgest,cdsegcli,tipcar,nuestce,tidadce,equipo,cdsegban,codprod,indvalo101,indvalo102,indvalo103,indvalo104,indvalo105,indvalo106,indvalo107,indvalo108,indvalo109,indvalo110,indvalo111,indvalo112,indvalo113,indvalo114,indvalo115,indvalo116,indvalo117,indvalo118,indvalo119,indvalo201,indvalo202,indvalo203,indvalo204,indvalo205,indvalo206,indvalo207,indvalo208,indvalo209,indvalo210,indvalo211,indvalo212,indvalo213,indvalo214,indvalo215,indvalo216,indvalo217,indvalo218,indvalo219,nivel,
rawDataSize=1571259, numRows=10000, serialization.lib=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
COLUMN_STATS_ACCURATE=true, totalSize=1581259, serialization.null.format=\N, transient_lastDdlTime=1482491985}
2016-12-29 11:39:40,299 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
Batch-Mode insert statement:
insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA"
(TIINDI
...
,NIVEL)
values
(:TIINDI
...
,:NIVEL)

2016-12-29 11:39:40,299 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
Prepared Statement SQL:
insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA"
(TIINDI
….
,:NIVEL)

2016-12-29 11:39:41,279 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatBase:
Prepared Statement SQL:
insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA"
(TIINDI
….
,:INDVALO219
,:NIVEL)

2016-12-29 11:39:41,679 ERROR [Thread-11] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler:
Thread Thread[Thread-11,5,main] threw an Exception.
java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12208)
	at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
	at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:231)
{code})

> Error exporting large table to Oracle and Append Values Hint (duplicated 3106)
> ------------------------------------------------------------------------------
>
>                 Key: SQOOP-3120
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3120
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.5
>            Reporter: Antonio Villar
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message