sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Controlling both transaction size and load during
Date Tue, 20 Sep 2011 20:35:40 GMT
During an import from a large table, we want to avoid using too many mappers, as that would
put too much load on the database.

However that winds up generating very large transactions, e.g. 30M+ rows per request.

Which in turn can cause a transaction timeout, if it takes longer than about 3000 seconds.

Is there any way to control both the load (number of parallel requests) and the size of each
request?

Thanks,

-- Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr




Mime
View raw message