Thoughts anyone?

Thanks,

Jason


On Tue, Mar 18, 2014 at 2:23 PM, Jason Rosenberg <jbr@squareup.com> wrote:
Hi,

I'm wondering if there is expected performance increases with using the --direct flag for exporting from hive to mysql.  If so, how much speedup?

Also, I've been getting lock contention errors during export, and I'm wondering if these are less likely using --direct mode?  E.g. I'm getting these sorts of exceptions on the sqoop console:

14/03/18 14:44:15 INFO mapred.JobClient: Task Id : attempt_201403180842_0202_m_000002_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.io.IOException: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
        at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWr


Thanks,

Jason