sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Garza <timothy.ga...@collinsongroup.com>
Subject RE: SQOOP v1.4.6 - sqoop export error
Date Fri, 19 Jun 2015 17:36:01 GMT
I noticed the text containing my resolution has gone missing.

This problem was caused by data-type mismatch between the HIVE table column (STRING) and the
target RDBMS column (CHAR, VARCHAR or TEXT). Basically Hive STRING will not convert naturally
into these ANSI data-types, I had to CAST the STRING into CHAR before the SQOOP Map-Reduce
would let me move the data across.

I noticed other people encountering this issue but nobody has posted a solution.

________________________________
Hi there

I had trouble with SQOOP export out of the box and have researched (ie. googled) this one
to death.

I wanted to publish the answer here for future generations as I myself couldn’t find a solution
and worked literally for hours with a colleague to investigate these errors.

I configured SQOOP locally successfully, which I confirmed by calling ‘sqoop version’:

[hadoop@ip-123-45-67-890 ~]$ sqoop version
15/06/19 09:07:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

Linux version:    Amazon Linux AMI release 2015.03

Syntax:
sqoop export --connect "jdbc:sqlserver://***URL GOES HERE***.us-west-2.rds.amazonaws.com:1433;databaseName=<databasename>"
--table <tablename> --export-dir hdfs://<IPgoeshere>:<port>/user/hive/warehouse/<hive_file>
--username <username> --password <password>

15/06/19 08:44:33 INFO mapreduce.Job: Task Id : attempt_1434676168391_0012_m_000001_0, Status
: FAILED
Error: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be cast to
org.apache.hadoop.io.LongWritable
        at org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
        at org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
        at org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:79)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:516)
        at org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:70)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:81)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)


Kind Regards

Timothy Garza


[Collinson Group]<http://www.collinsongroup.com/>




The Collinson Group Limited; Registered number: 2577557, Registered in England & Wales;
Registered Office: Cutlers Exchange, 123 Houndsditch, London, EC3A 7BU.


This e-mail may contain privileged and confidential information and/or copyright material
and is intended for the use of the addressee only. If you receive this e-mail by mistake please
advise the sender immediately by using the reply facility in your e-mail software and delete
this e-mail from your computer system. You may not deliver, copy or disclose its contents
to anyone else. Any unauthorised use may be unlawful. Any views expressed in this e-mail are
those of the individual sender and may not necessarily reflect the views of The Collinson
Group Ltd and/or its subsidiaries or any other associated company (collectively “Collinson
Group”).

As communications via the Internet are not secure Collinson Group cannot accept any liability
if this e-mail is accessed by third parties during the course of transmission or is modified
or amended in any way following despatch. Collinson Group cannot guarantee that any attachment
to this email does not contain a virus, therefore it is strongly recommended that you carry
out your own virus check before opening any attachment, as we cannot accept liability for
any damage sustained as a result of software virus infection. Senders of messages shall be
taken to consent to the monitoring and recording of e-mails addressed to members of the Company.
Mime
View raw message