sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leonardo Brambilla <lbrambi...@contractor.elance-odesk.com>
Subject Fwd: Sqoop export not working when using "update-key"
Date Sun, 11 May 2014 13:33:11 GMT
Hello, I am struggling to make it work, what is a really required feature.

I have a process that daily generates new data, this data needs to be
pushed to a table in Oracle, the table might already have same data from
previous loads. I need to avoid duplicating data on it. Pretty common
scenario right? =)

I am using sqoop export for this, no special arguments, just columns,
fields-terminated-by, table and db connection, plus the argument
"update-mode allowinsert".

Now, when I also include the argument "update-key" with a comma separated
list of fields (which is the same for arg columns) I get the following
oracle driver error:

14/05/07 16:00:03 INFO mapred.JobClient: Task Id :
attempt_201404190827_0928_m_000003_2, Status : FAILED
java.io.IOException: Can't export data, please check task tracker logs
        at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
        at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.sql.SQLException: Missing IN or OUT
parameter at index:: 4
        at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
        at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
        at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:639)
        at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:84)
        ... 10 more
*Caused by: java.sql.SQLException: Missing IN or OUT parameter at index:: 4*
        at
oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844)
        at
oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213)
        at
oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362)
        at
org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174)
        at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
        at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:215)
        ... 14 more

I'm using Sqoop 1.4.3 with hadoop1, also tried 1.4.4 with same result. I
have the standard Oracle JDBC driver 6 with Java 7.
I went through all the documentation, Sqoop user guide says this is
supported for built-in connector which I understand I am using.

Here is the full command:
$sqoopExecutable export \
--outdir $outdir \
--connect $connectionString --table $table_client --username $dbUser
--password $dbUserPasswd \
--columns CLIENT_ID,EXP_ID,BUCKET_ID --update-key
CLIENT_ID,EXP_ID,BUCKET_ID \
--fields-terminated-by '\t' --update-mode allowinsert \
--export-dir $dataSource_client > $sqoopLog 2>&1

Can someone please shed some light on this?
Thank you in advance.

Leo

Mime
View raw message