sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Fwd: sqoop and --direct option
Date Tue, 02 Apr 2013 17:56:39 GMT
Hi Thuy,
in case of direct import all your data are passed directly to mysqlimport utility that is
then doing all the parsing and inserting into your database. Without direct mode Sqoop itself
will parse data so that they can be inserted. Thus slight differences in edge error cases
are expected as there are different tools involved to process the data.

Sqoop by default assumes that you're using string constant "null" to encode NULL values and
everything else is assumed to be proper content of the column. However Hive is using different
string constant "\N" to encode the NULL value. You need to tell Sqoop what is the correct
substitution constant using --input-(non-)string parameters, like:

  sqoop export ... --input-null-string '\\N' --input-null-non-string '\\N'

Please note that the double backslash is needed due to way how this parameter is used withing
Sqoop. Please check out Sqoop user guide if you have further questions [1].


1: http://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_importing_data_into_hive

On Tue, Apr 02, 2013 at 10:34:52AM -0700, Thuy Nguyen wrote:
> Hi all,
> I got error exporting from hive to mysql (without --direct option) as below:
> java.lang.NumberFormatException: For input string: "\N"
>         at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>         at java.lang.Long.parseLong(Long.java:410)
>         at java.lang.Long.valueOf(Long.java:525)
>         at
> hive_category_event_type_facts.__loadFromFields(hive_category_event_type_facts.java:314)
>         at
> hive_category_event_type_facts.parse(hive_category_event_type_facts.java:251)
>         at
> com.cloudera.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:81)
>         at
> com.cloudera.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:40)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at
> com.cloudera.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:189)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.had
> and by running swoop with --direct option, '\N' is converted to 0 and thus,
> sqoop succeeded.
> Could you please advise if --direct option is expected to have that side
> effect?
> Thanks,
> Thuy

View raw message