sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Fwd: Sqoop export not working when using "update-key"
Date Tue, 13 May 2014 01:44:28 GMT
The map task log contain entire executed query and lot of additional information and hence
it's supper useful in such cases.

Jarcec

On Mon, May 12, 2014 at 02:59:56PM -0300, Leonardo Brambilla wrote:
> Hi Jarek,
> 
> thanks for replying, I don't have the logs. I'll see if I can run the task
> again and then keep the logs.
> 
> Anyway, I don't remember seeing anything else than this SQLException about
> missing parameter.
> 
> Leo
> 
> 
> On Sun, May 11, 2014 at 10:59 AM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:
> 
> > Hi Leonardo,
> > would you mind sharing with us task log from the failed map task?
> >
> > Jarcec
> >
> > On Sun, May 11, 2014 at 10:33:11AM -0300, Leonardo Brambilla wrote:
> > > Hello, I am struggling to make it work, what is a really required
> > feature.
> > >
> > > I have a process that daily generates new data, this data needs to be
> > > pushed to a table in Oracle, the table might already have same data from
> > > previous loads. I need to avoid duplicating data on it. Pretty common
> > > scenario right? =)
> > >
> > > I am using sqoop export for this, no special arguments, just columns,
> > > fields-terminated-by, table and db connection, plus the argument
> > > "update-mode allowinsert".
> > >
> > > Now, when I also include the argument "update-key" with a comma separated
> > > list of fields (which is the same for arg columns) I get the following
> > > oracle driver error:
> > >
> > > 14/05/07 16:00:03 INFO mapred.JobClient: Task Id :
> > > attempt_201404190827_0928_m_000003_2, Status : FAILED
> > > java.io.IOException: Can't export data, please check task tracker logs
> > >         at
> > >
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
> > >         at
> > > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> > >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > >         at
> > >
> > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> > >         at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > >         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > >         at java.security.AccessController.doPrivileged(Native Method)
> > >         at javax.security.auth.Subject.doAs(Subject.java:415)
> > >         at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> > >         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > > Caused by: java.io.IOException: java.sql.SQLException: Missing IN or OUT
> > > parameter at index:: 4
> > >         at
> > >
> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
> > >         at
> > >
> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
> > >         at
> > >
> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:639)
> > >         at
> > >
> > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > >         at
> > > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:84)
> > >         ... 10 more
> > > *Caused by: java.sql.SQLException: Missing IN or OUT parameter at
> > index:: 4*
> > >         at
> > >
> > oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844)
> > >         at
> > >
> > oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213)
> > >         at
> > >
> > oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362)
> > >         at
> > >
> > org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174)
> > >         at
> > >
> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> > >         at
> > >
> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:215)
> > >         ... 14 more
> > >
> > > I'm using Sqoop 1.4.3 with hadoop1, also tried 1.4.4 with same result. I
> > > have the standard Oracle JDBC driver 6 with Java 7.
> > > I went through all the documentation, Sqoop user guide says this is
> > > supported for built-in connector which I understand I am using.
> > >
> > > Here is the full command:
> > > $sqoopExecutable export \
> > > --outdir $outdir \
> > > --connect $connectionString --table $table_client --username $dbUser
> > > --password $dbUserPasswd \
> > > --columns CLIENT_ID,EXP_ID,BUCKET_ID --update-key
> > > CLIENT_ID,EXP_ID,BUCKET_ID \
> > > --fields-terminated-by '\t' --update-mode allowinsert \
> > > --export-dir $dataSource_client > $sqoopLog 2>&1
> > >
> > > Can someone please shed some light on this?
> > > Thank you in advance.
> > >
> > > Leo
> >

Mime
View raw message