sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wenzhi wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1037) Cannot import mysql table with into Hadoop
Date Tue, 21 May 2013 02:35:17 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662600#comment-13662600
] 

wenzhi wang commented on SQOOP-1037:
------------------------------------

Jecec,

Thanks, I opened #SQOOP-1054 to request Sqoop to support binary type splitter. Although there
are integer columns in the table, I cannot define key on them.

Another question: I also tried import table by "--split-by currentMapScreeningProfile" and
without "-m 1". This time, It costs 219s to import same amount of data. Currently, field "currentMapScreeningProfile"
is NULL for all records:

mysql> SELECT MIN(`currentMapScreeningProfile`), MAX(`currentMapScreeningProfile`) FROM
`IMSI`;
+-----------------------------------+-----------------------------------+
| MIN(`currentMapScreeningProfile`) | MAX(`currentMapScreeningProfile`) |
+-----------------------------------+-----------------------------------+
|                              NULL |                              NULL |
+-----------------------------------+-----------------------------------+
1 row in set (19.64 sec)

Why the import duation shortens amost a half? Does Sqoop import table with more than 1 mapper
even when splliter's values are same?

Thanks,
Wenzhi





                
> Cannot import mysql table with into Hadoop
> ------------------------------------------
>
>                 Key: SQOOP-1037
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1037
>             Project: Sqoop
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 1.4.3
>         Environment: bl1253{wenzhiw}/home/wenzhiw>cat /etc/redhat-release
> Scientific Linux SL release 5.3 (Boron)
> bl1253{wenzhiw}/home/wenzhiw>uname -a
> Linux bl1253.cn.alcatel-lucent.com 2.6.18-128.7.1.el5PAE #1 SMP Mon Aug 24 08:41:30 EDT
2009 i686 athlon i386 GNU/Linux
>            Reporter: wenzhi wang
>            Priority: Critical
>         Attachments: sqoop_import_error.log
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Hi,
> I run Sqoop 1.4.3 on Apache Hadoop 1.0.4 and want to import table from mysql, but met
below errors. I also searched jira site for this issue, but didn't find any useful info, please
help to check. thanks!
> {code}
> bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>sqoop import
--table IMSI --connect jdbc:mysql://135.252.158.250/hlr --username root -P
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> Warning: $HADOOP_HOME is deprecated.
> Enter password:
> 13/05/08 10:20:53 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 13/05/08 10:20:53 INFO tool.CodeGenTool: Beginning code generation
> 13/05/08 10:20:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `IMSI`
AS t LIMIT 1
> 13/05/08 10:20:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `IMSI`
AS t LIMIT 1
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> 13/05/08 10:20:55 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
> ...
> {code}
> Table definition:
> {code}
> mysql> desc IMSI;
> +-------------------------------+-------------+------+-----+---------+-------+
> | Field                         | Type        | Null | Key | Default | Extra |
> +-------------------------------+-------------+------+-----+---------+-------+
> | mSubIdentificationNumberId    | binary(6)   | NO   | PRI | NULL    |       |
> | mobileNetworkCodeId           | binary(3)   | NO   | PRI | NULL    |       |
> | mobileCountryCodeId           | binary(3)   | NO   | PRI | NULL    |       |
> | mainMSISDN                    | binary(12)  | YES  |     | NULL    |       |
> | alertMSISDN                   | binary(12)  | YES  |     | NULL    |       |
> | associatedMSISDN              | binary(88)  | YES  |     | NULL    |       |
> | gprsNotReachableReason        | tinyint(4)  | YES  |     | NULL    |       |
> | sgsnLastGlobalUpdateTime      | int(11)     | YES  |     | NULL    |       |
> | vlrIsdn                       | binary(10)  | YES  |     | NULL    |       |
> | mscIsdn                       | binary(10)  | YES  |     | NULL    |       |
> | supOfLocRegCount              | tinyint(4)  | YES  |     | NULL    |       |
> | locRegSubsPeriod              | smallint(6) | YES  |     | NULL    |       |
> | inducedCbocBsg                | binary(7)   | YES  |     | NULL    |       |
> | inducedCboiBsg                | binary(7)   | YES  |     | NULL    |       |
> | inducedCboehBsg               | binary(7)   | YES  |     | NULL    |       |
> | vlrLastGlobalUpdateTime       | int(11)     | YES  |     | NULL    |       |
> | currentMapScreeningProfile    | int(11)     | YES  |     | NULL    |       |
> | smscIsdn                      | binary(70)  | YES  |     | NULL    |       |
> | gsmNotReachableReason         | tinyint(4)  | YES  |     | NULL    |       |
> | timeStampWaitingSms           | binary(28)  | YES  |     | NULL    |       |
> | checkSSindicator              | tinyint(4)  | YES  |     | NULL    |       |
> | supOfAutReqCount              | tinyint(4)  | YES  |     | NULL    |       |
> | autReqSubsPeriod              | smallint(6) | YES  |     | NULL    |       |
> | underTestImsi                 | bit(1)      | YES  |     | NULL    |       |
> | cardStatus                    | bit(1)      | YES  |     | NULL    |       |
> | 
> +-------------------------------+-------------+------+-----+---------+-------+
> {code}
> Thanks,
> Wenzhi

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message