sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: direct mode for mysql export vs. default jdbc
Date Thu, 20 Mar 2014 15:24:18 GMT
Thoughts anyone?

Thanks,

Jason


On Tue, Mar 18, 2014 at 2:23 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> Hi,
>
> I'm wondering if there is expected performance increases with using the
> --direct flag for exporting from hive to mysql.  If so, how much speedup?
>
> Also, I've been getting lock contention errors during export, and I'm
> wondering if these are less likely using --direct mode?  E.g. I'm getting
> these sorts of exceptions on the sqoop console:
>
> 14/03/18 14:44:15 INFO mapred.JobClient: Task Id :
> attempt_201403180842_0202_m_000002_1, Status : FAILED
> java.io.IOException: Can't export data, please check failed map task logs
>         at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
>         at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
>         at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>         at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.io.IOException: java.sql.BatchUpdateException: Deadlock
> found when trying to get lock; try restarting transaction
>         at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWr
>
>
> Thanks,
>
> Jason
>

Mime
View raw message