sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Hall <Peter.H...@quest.com>
Subject RE: [sqoop-user] Support for partitioning during export into HDFS
Date Thu, 01 Sep 2011 23:52:02 GMT
Hi Ken,

We did initially consider an approach similar to what you suggest, but decided not to go with
it due to complexities when the number of mappers is different to the number of partitions.
Instead we are breaking up the blocks in the table and spreading them across all the mappers
and doing ROWID range scans. So all mappers could be reading from all partitions - but they
would only be reading part of each. Splitting by PARTITION may provide slightly better performance,
but we don't believe it would be a huge difference.

Regards,
Peter Hall
Quest Software
Mime
View raw message