sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Xie <xie3208...@gmail.com>
Subject sqoop import big table with no integer key
Date Wed, 11 Jul 2018 14:05:38 GMT
Good day,

In my situation my table has billion rows, it doesn't come with an integer
column as its key, that means if I use sqoop to do the import (into hive),
I would not be able to use multiple mapper.
As table's size is big, it is not realistic to add an extra new integer
field to it.

I do come across a post from hortonworks which seems to suggest it is
possible however was commented that:

1. no guarantees though that sqoop splits your records evenly over your
mappers though.
2. For huge number of row the above options will cause duplicates in the
results set.


Any thought?

Thank you very much.

*Sincerely yours,*


View raw message