sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brenden Cobb <Brenden.C...@humedica.com>
Subject Re: Sqoop to Oracle transfer rates
Date Thu, 19 Jun 2014 14:44:13 GMT
Thanks David that is very informative. May I ask what partitioning method you utilized?

I've got Oracle licensing covered so that should not be an issue.

From: David Robson <David.Robson@software.dell.com<mailto:David.Robson@software.dell.com>>
Reply-To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Date: Wednesday, June 18, 2014 7:43 PM
To: "user@sqoop.apache.org<mailto:user@sqoop.apache.org>" <user@sqoop.apache.org<mailto:user@sqoop.apache.org>>
Subject: RE: Sqoop to Oracle transfer rates

Hi Brenden,

I did some benchmarking a while ago on a 62858MB folder. The best I achieved with 24 mappers
was 867 seconds (72.5MB/s). This was on a 4 node Hadoop cluster that was on some old servers
we had so I’m sure with some better hardware you could get a lot faster.

I’ll attach some graphs which you might find interesting – basically you can see OraOop
reduces load on the DB a little bit with a few optimizations – but once you use direct path
and partition exchange loading the benefits are massive. Of course this requires you to have
the relevant Oracle licenses so may not be an option.

David

From: Brenden Cobb [mailto:Brenden.Cobb@humedica.com]
Sent: Thursday, 19 June 2014 6:25 AM
To: user@sqoop.apache.org<mailto:user@sqoop.apache.org>
Subject: Sqoop to Oracle transfer rates

Looking for some benchmarks on Sqoop (or Oraoop) exports.

Appreciate if anyone feels like sharing some metrics. Rough numbers for MB/s would suffice,
perhaps number of mappers used. I'm working specifically with Oracle exports, but other RDBMS
export rates would be enlightening as well.

Thanks

Mime
View raw message