sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject direct mode for mysql export vs. default jdbc
Date Tue, 18 Mar 2014 21:23:36 GMT
Hi,

I'm wondering if there is expected performance increases with using the
--direct flag for exporting from hive to mysql.  If so, how much speedup?

Also, I've been getting lock contention errors during export, and I'm
wondering if these are less likely using --direct mode?  E.g. I'm getting
these sorts of exceptions on the sqoop console:

14/03/18 14:44:15 INFO mapred.JobClient: Task Id :
attempt_201403180842_0202_m_000002_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
        at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
        at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
        at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.io.IOException: java.sql.BatchUpdateException: Deadlock
found when trying to get lock; try restarting transaction
        at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWr


Thanks,

Jason

Mime
View raw message