sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei Yan <ywsk...@gmail.com>
Subject Does Sqoop support small queries?
Date Wed, 29 Jun 2016 17:35:49 GMT

Would like to check whether Sqoop supports this type of ingestion: consider
we have records with range [1,12], and we have 3 mappers. So in default,
the 3 mappers will be assigned [1,4], [5, 8], [9, 12].

Not sure whether we can split the range to smaller one, like, [1], [2],
[3], ..., [12]. But still using 3 mappers instead of 12 mappers. We want
this feature because: (1) if configured smaller mapper number, each mapper
will be assigned a larger range and take much longer time to finish, and
the infra may kill long running query; (2) But if we configured a larger
mapper number, each mapper has a smaller range, but meanwhile we generates
lots of network traffic to the database, which will also be bad. One good
way we want is: still 12 ranges, but 3 mappers, and at most 3 concurrent
connections at most.

Appreciate any help here.


View raw message