sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kate Ting <k...@cloudera.com>
Subject Re: [sqoop-user] Problem using sqoop with --direct (mysqldump)
Date Tue, 20 Sep 2011 01:15:41 GMT
[To: sqoop-user@incubator.apache.org; Bcc: sqoop-user@cloudera.org]

Eric - to rule out permission issues and mysqldump issues:

1. Does username 'sqoop' have permission to access table1?
2. Can you ssh into one of the node machines, run mysqldump, and then
paste the output?

Regards, Kate

On Sat, Sep 17, 2011 at 1:05 PM, eric hernandez <eric.hardway@gmail.com> wrote:
> Kate,
> I am sqooping from a MySQL Server version: 5.0.58-enterprise-gpl MySQL
> Enterprise Server
>
> Currently i am using on all data nodes
> mysqldump  Ver 10.13 Distrib 5.5.15, for Linux (x86_64)
>
> But I tried also using
> mysql-5.0.77-4.el5_6.6.x86_64   mysqldump from the centos 5.5 repo.
>
> Both fail.
>
> -Eric
>
> On Sat, Sep 17, 2011 at 11:43 AM, Kate Ting <kate@cloudera.com> wrote:
>>
>> Eric, what is your mysql version and your mysqldump version?
>>
>> Regards, Kate
>>
>> On Fri, Sep 16, 2011 at 7:30 PM, eric hernandez <eric.hardway@gmail.com>
>> wrote:
>> > Kate,
>> >
>> > sqoop import --connect jdbc:mysql://192.168.0.100:3307/test --verbose -m
>> > 1
>> > --username sqoop --password sanitized --hive-overwrite --direct --table
>> > table1 --hive-import --create-hive-table --hive-table table1
>> > --fields-terminated-by '\t' --lines-terminated-by '\n' --append
>> >
>> > Please note this works fine if i remove the --direct option. I am also
>> > limiting it to 1 mapper because if not the output of the failure is very
>> > long.
>> >
>> >
>> > On Fri, Sep 16, 2011 at 3:01 PM, Kate Ting <kate@cloudera.com> wrote:
>> >>
>> >> Eric - what is the exact Sqoop command that you ran (including, if
>> >> applicable, contents of the options-file)?
>> >>
>> >> Regards, Kate
>> >>
>> >> On Fri, Sep 16, 2011 at 9:27 AM, eric hernandez
>> >> <eric.hardway@gmail.com>
>> >> wrote:
>> >> > Yes i have mysqldump on all nodes.
>> >> >
>> >> > Verbose output
>> >> >
>> >> >
>> >> >
>> >> > 11/09/16 09:22:42 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>> >> > 11/09/16 09:22:42 WARN tool.BaseSqoopTool: Setting your password on
>> >> > the
>> >> > command-line is insecure. Consider using -P instead.
>> >> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Loaded manager factory:
>> >> > com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>> >> > com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> > 11/09/16 09:22:42 DEBUG manager.DefaultManagerFactory: Trying with
>> >> > scheme:
>> >> > jdbc:mysql:
>> >> > 11/09/16 09:22:42 INFO manager.MySQLManager: Preparing to use a MySQL
>> >> > streaming resultset.
>> >> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>> >> > com.cloudera.sqoop.manager.DirectMySQLManager@7ad81784
>> >> > 11/09/16 09:22:42 INFO tool.CodeGenTool: Beginning code generation
>> >> > 11/09/16 09:22:42 DEBUG manager.SqlManager: No connection paramenters
>> >> > specified. Using regular API for making connection.
>> >> > 11/09/16 09:22:43 DEBUG manager.SqlManager: Using fetchSize for next
>> >> > query:
>> >> > -2147483648
>> >> > 11/09/16 09:22:43 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT
>> >> > t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 09:22:43 DEBUG manager.SqlManager: Using fetchSize for next
>> >> > query:
>> >> > -2147483648
>> >> > 11/09/16 09:22:43 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT
>> >> > t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: selected columns:
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   id
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   application_id
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   event_id
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   response_id
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   target_id
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   mode
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   date_created
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Writing source file:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.java
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Table name: table1
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Columns: id:4,
>> >> > application_id:4,
>> >> > event_id:4, response_id:4, target_id:4, mode:1, date_created:93,
>> >> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: sourceFilename is
>> >> > table1.java
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Found existing
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> >> > 11/09/16 09:22:43 INFO orm.CompilationManager: HADOOP_HOME is
>> >> > /usr/lib/hadoop
>> >> > 11/09/16 09:22:43 INFO orm.CompilationManager: Found hadoop core jar
>> >> > at:
>> >> > /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Adding source file:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.java
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Invoking javac with
>> >> > args:
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -sourcepath
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -d
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -classpath
>> >> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> >> >
>> >> >
>> >> > /etc/hadoop/conf:/usr/java/jdk1.6.0_21/lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/hadoop-fairscheduler-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/lib/hue-plugins-1.2.0-cdh3u1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/junit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.2.2.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/sqoop/conf:/etc/zookeeper::/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar:/usr/lib/sqoop/lib/avro-1.5.1.jar:/usr/lib/sqoop/lib/avro-ipc-1.5.1.jar:/usr/lib/sqoop/lib/avro-mapred-1.5.1.jar:/usr/lib/sqoop/lib/commons-io-1.4.jar:/usr/lib/sqoop/lib/ivy-2.0.0-rc2.jar:/usr/lib/sqoop/lib/jackson-core-asl-1.7.3.jar:/usr/lib/sqoop/lib/jackson-mapper-asl-1.7.3.jar:/usr/lib/sqoop/lib/jopt-simple-3.2.jar:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar:/usr/lib/sqoop/lib/paranamer-2.3.jar:/usr/lib/sqoop/lib/snappy-java-1.0.3-rc2.jar:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar:/usr/lib/sqoop/sqoop-test-1.3.0-cdh3u1.jar::/usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> >> > 11/09/16 09:22:44 INFO orm.CompilationManager: Writing jar file:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.jar
>> >> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Scanning for .class
>> >> > files in
>> >> > directory: /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b
>> >> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Got classfile:
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.class
>> >> > ->
>> >> > table1.class
>> >> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Finished writing jar
>> >> > file
>> >> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.jar
>> >> > 11/09/16 09:22:44 DEBUG tool.ImportTool: Using temporary folder:
>> >> > 16092244516table1
>> >> > 11/09/16 09:22:44 INFO manager.DirectMySQLManager: Beginning
>> >> > mysqldump
>> >> > fast
>> >> > path import
>> >> > 11/09/16 09:22:44 INFO mapreduce.ImportJobBase: Beginning import of
>> >> > table1
>> >> > 11/09/16 09:22:44 DEBUG manager.SqlManager: Using fetchSize for next
>> >> > query:
>> >> > -2147483648
>> >> > 11/09/16 09:22:44 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT
>> >> > t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.MySQLDumpImportJob: Using
>> >> > InputFormat:
>> >> > class com.cloudera.sqoop.mapreduce.MySQLDumpInputFormat
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/jackson-mapper-asl-1.7.3.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/snappy-java-1.0.3-rc2.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/paranamer-2.3.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/ivy-2.0.0-rc2.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/jackson-core-asl-1.7.3.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/avro-mapred-1.5.1.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/avro-1.5.1.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/jopt-simple-3.2.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/commons-io-1.4.jar
>> >> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> >> > file:/usr/lib/sqoop/lib/avro-ipc-1.5.1.jar
>> >> > 11/09/16 09:22:46 INFO mapred.JobClient: Running job:
>> >> > job_201109160744_0006
>> >> > 11/09/16 09:22:47 INFO mapred.JobClient:  map 0% reduce 0%
>> >> > 11/09/16 09:22:53 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0006_m_000000_0, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >     at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >     at java.security.AccessController.doPrivileged(Native Method)
>> >> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >     at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0006_m_000000_0: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0006_m_000000_0:     at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0006_m_000000_0:     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0006_m_000000_0: log4j:WARN No appenders could
>> >> > be
>> >> > found
>> >> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0006_m_000000_0: log4j:WARN Please initialize
>> >> > the
>> >> > log4j
>> >> > system properly.
>> >> > 11/09/16 09:22:58 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0006_m_000000_1, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >     at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >     at java.security.AccessController.doPrivileged(Native Method)
>> >> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >     at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0006_m_000000_1: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0006_m_000000_1:     at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0006_m_000000_1:     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0006_m_000000_1: log4j:WARN No appenders could
>> >> > be
>> >> > found
>> >> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0006_m_000000_1: log4j:WARN Please initialize
>> >> > the
>> >> > log4j
>> >> > system properly.
>> >> > 11/09/16 09:23:03 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0006_m_000000_2, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >> >     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >     at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >     at java.security.AccessController.doPrivileged(Native Method)
>> >> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >     at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0006_m_000000_2: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0006_m_000000_2:     at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0006_m_000000_2:     at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0006_m_000000_2: log4j:WARN No appenders could
>> >> > be
>> >> > found
>> >> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0006_m_000000_2: log4j:WARN Please initialize
>> >> > the
>> >> > log4j
>> >> > system properly.
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient: Job complete:
>> >> > job_201109160744_0006
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient: Counters: 6
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:   Job Counters
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=19196
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     Total time spent by
all
>> >> > reduces
>> >> > waiting after reserving slots (ms)=0
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     Total time spent by
all
>> >> > maps
>> >> > waiting after reserving slots (ms)=0
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     Launched map tasks=4
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
>> >> > 11/09/16 09:23:09 INFO mapred.JobClient:     Failed map tasks=1
>> >> > 11/09/16 09:23:09 INFO mapreduce.ImportJobBase: Transferred 0 bytes
>> >> > in
>> >> > 24.8354 seconds (0 bytes/sec)
>> >> > 11/09/16 09:23:09 INFO mapreduce.ImportJobBase: Retrieved 0 records.
>> >> > 11/09/16 09:23:09 ERROR tool.ImportTool: Error during import: Import
>> >> > job
>> >> > failed!
>> >> >
>> >> >
>> >> > --- Task log
>> >> >
>> >> > Task Logs: 'attempt_201109160744_0006_m_000000_1'
>> >> >
>> >> > stdout logs
>> >> > ________________________________
>> >> >
>> >> > stderr logs
>> >> >
>> >> > Exception in thread "Thread-12" java.lang.
>> >> > IndexOutOfBoundsException
>> >> >       at java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> >       at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > log4j:WARN No appenders could be found for logger
>> >> > (org.apache.hadoop.hdfs.DFSClient).
>> >> > log4j:WARN Please initialize the log4j system properly.
>> >> >
>> >> > ________________________________
>> >> >
>> >> > syslog logs
>> >> >
>> >> > 2011-09-16 09:22:54,194 WARN org.apache.hadoop.util.NativeCodeLoader:
>> >> > Unable
>> >> > to load native-hadoop library for your platform... using builtin-java
>> >> > classes where applicable
>> >> > 2011-09-16 09:22:54,326 INFO
>> >> > org.apache.hadoop.metrics.jvm.JvmMetrics:
>> >> > Initializing JVM Metrics with processName=MAP, sessionId=
>> >> > 2011-09-16 09:22:54,687 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > Beginning mysqldump fast path import
>> >> > 2011-09-16 09:22:54,690 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > Performing import of table table1 from database test
>> >> > 2011-09-16 09:22:54,696 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > Converting data to use specified delimiters.
>> >> > 2011-09-16 09:22:54,696 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > (For the fastest possible import, use
>> >> > 2011-09-16 09:22:54,696 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > --mysql-delimiters to specify the same field
>> >> > 2011-09-16 09:22:54,696 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > delimiters as are used by mysqldump.)
>> >> > 2011-09-16 09:22:54,710 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > mysqldump: Got errno 32 on write
>> >> > 2011-09-16 09:22:54,710 INFO
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> >> > Transfer loop complete.
>> >> > 2011-09-16 09:22:54,740 INFO
>> >> > org.apache.hadoop.mapred.TaskLogsTruncater:
>> >> > Initializing logs' truncater with mapRetainSize=-1 and
>> >> > reduceRetainSize=-1
>> >> > 2011-09-16 09:22:54,746 WARN org.apache.hadoop.mapred.Child: Error
>> >> > running
>> >> > child
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >       at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >> >       at
>> >> >
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >> >       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >       at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >       at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >       at java.security.AccessController.doPrivileged(Native Method)
>> >> >       at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >       at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >> >       at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> > 2011-09-16 09:22:54,750 INFO org.apache.hadoop.mapred.Task: Runnning
>> >> > cleanup
>> >> > for the task
>> >> >
>> >> >
>> >> > On Fri, Sep 16, 2011 at 8:55 AM, Kate Ting <kate@cloudera.com>
wrote:
>> >> >>
>> >> >> [Moving the conversation to sqoop-user@incubator.apache.org. Please
>> >> >> subscribe (and post questions) to the new mailing list.]
>> >> >>
>> >> >> Hi Eric -
>> >> >>
>> >> >> (1) Is the mysqldump utility installed on individual node machines?
>> >> >> (2) If so, can you pastebin your task log as well as verbose output?
>> >> >>
>> >> >> Regards, Kate
>> >> >>
>> >> >> On Fri, Sep 16, 2011 at 8:04 AM, Eric <eric.hardway@gmail.com>
>> >> >> wrote:
>> >> >> > Hi all,
>> >> >> >
>> >> >> > I cannot sqoop in using the --direct option, my sqoop works
fine
>> >> >> > up
>> >> >> > until i add --direct .
>> >> >> >
>> >> >> > I am using Sqoop 1.3.0-cdh3u1
>> >> >> > git commit id 3a60cc809b14d538dd1eb0e90ffa9767e8d06a43
>> >> >> > Compiled by jenkins@ubuntu-slave01 on Mon Jul 18 08:38:49
PDT 2011
>> >> >> >
>> >> >> > Please Advise,
>> >> >> >
>> >> >> > -Eric
>> >> >> >
>> >> >> >
>> >> >> > error message:
>> >> >> >
>> >> >> > 11/09/16 07:57:39 INFO manager.MySQLManager: Preparing to
use a
>> >> >> > MySQL
>> >> >> > streaming resultset.
>> >> >> > 11/09/16 07:57:39 INFO tool.CodeGenTool: Beginning code generation
>> >> >> > 11/09/16 07:57:40 INFO manager.SqlManager: Executing SQL
>> >> >> > statement:
>> >> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> >> > 11/09/16 07:57:40 INFO manager.SqlManager: Executing SQL
>> >> >> > statement:
>> >> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> >> > 11/09/16 07:57:40 INFO orm.CompilationManager: HADOOP_HOME
is
>> >> >> > /usr/lib/
>> >> >> > hadoop
>> >> >> > 11/09/16 07:57:40 INFO orm.CompilationManager: Found hadoop
core
>> >> >> > jar
>> >> >> > at: /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar
>> >> >> > 11/09/16 07:57:41 INFO orm.CompilationManager: Writing jar
file:
>> >> >> > /tmp/
>> >> >> > sqoop-root/compile/aef5c62d2156aeae5338ee272de42d26/table1.jar
>> >> >> > 11/09/16 07:57:41 INFO manager.DirectMySQLManager: Beginning
>> >> >> > mysqldump
>> >> >> > fast path import
>> >> >> > 11/09/16 07:57:41 INFO mapreduce.ImportJobBase: Beginning
import
>> >> >> > of
>> >> >> > table1
>> >> >> > 11/09/16 07:57:41 INFO manager.SqlManager: Executing SQL
>> >> >> > statement:
>> >> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> >> > 11/09/16 07:57:43 INFO mapred.JobClient: Running job:
>> >> >> > job_201109160744_0004
>> >> >> > 11/09/16 07:57:44 INFO mapred.JobClient:  map 0% reduce 0%
>> >> >> > 11/09/16 07:57:50 INFO mapred.JobClient: Task Id :
>> >> >> > attempt_201109160744_0004_m_000000_0, Status : FAILED
>> >> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 476)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 49)
>> >> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >> >        at
>> >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >> >        at java.security.AccessController.doPrivileged(Native
>> >> >> > Method)
>> >> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> >> > 1127)
>> >> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >> >
>> >> >> > attempt_201109160744_0004_m_000000_0: Exception in thread
>> >> >> > "Thread-12"
>> >> >> > java.lang.IndexOutOfBoundsException
>> >> >> > attempt_201109160744_0004_m_000000_0:   at
>> >> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> >> > attempt_201109160744_0004_m_000000_0:   at
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> >> > attempt_201109160744_0004_m_000000_0: log4j:WARN No appenders
>> >> >> > could
>> >> >> > be
>> >> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> >> > attempt_201109160744_0004_m_000000_0: log4j:WARN Please initialize
>> >> >> > the
>> >> >> > log4j system properly.
>> >> >> > 11/09/16 07:57:55 INFO mapred.JobClient: Task Id :
>> >> >> > attempt_201109160744_0004_m_000000_1, Status : FAILED
>> >> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 476)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 49)
>> >> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >> >        at
>> >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >> >        at java.security.AccessController.doPrivileged(Native
>> >> >> > Method)
>> >> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> >> > 1127)
>> >> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >> >
>> >> >> > attempt_201109160744_0004_m_000000_1: Exception in thread
>> >> >> > "Thread-12"
>> >> >> > java.lang.IndexOutOfBoundsException
>> >> >> > attempt_201109160744_0004_m_000000_1:   at
>> >> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> >> > attempt_201109160744_0004_m_000000_1:   at
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> >> > attempt_201109160744_0004_m_000000_1: log4j:WARN No appenders
>> >> >> > could
>> >> >> > be
>> >> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> >> > attempt_201109160744_0004_m_000000_1: log4j:WARN Please initialize
>> >> >> > the
>> >> >> > log4j system properly.
>> >> >> > 11/09/16 07:58:01 INFO mapred.JobClient: Task Id :
>> >> >> > attempt_201109160744_0004_m_000000_2, Status : FAILED
>> >> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 476)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> >> > 49)
>> >> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >> >        at
>> >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >> >        at java.security.AccessController.doPrivileged(Native
>> >> >> > Method)
>> >> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >> >        at
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> >> > 1127)
>> >> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >> >
>> >> >> > attempt_201109160744_0004_m_000000_2: Exception in thread
>> >> >> > "Thread-12"
>> >> >> > java.lang.IndexOutOfBoundsException
>> >> >> > attempt_201109160744_0004_m_000000_2:   at
>> >> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> >> > attempt_201109160744_0004_m_000000_2:   at
>> >> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> >> > attempt_201109160744_0004_m_000000_2: log4j:WARN No appenders
>> >> >> > could
>> >> >> > be
>> >> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> >> > attempt_201109160744_0004_m_000000_2: log4j:WARN Please initialize
>> >> >> > the
>> >> >> > log4j system properly.
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient: Job complete:
>> >> >> > job_201109160744_0004
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient: Counters: 6
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:   Job Counters
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:
>> >> >> > SLOTS_MILLIS_MAPS=19165
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Total time
spent by
>> >> >> > all
>> >> >> > reduces waiting after reserving slots (ms)=0
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Total time
spent by
>> >> >> > all
>> >> >> > maps waiting after reserving slots (ms)=0
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Launched map
tasks=4
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:
>> >> >> > SLOTS_MILLIS_REDUCES=0
>> >> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Failed map
tasks=1
>> >> >> > 11/09/16 07:58:07 INFO mapreduce.ImportJobBase: Transferred
0
>> >> >> > bytes
>> >> >> > in
>> >> >> > 25.1844 seconds (0 bytes/sec)
>> >> >> > 11/09/16 07:58:07 INFO mapreduce.ImportJobBase: Retrieved
0
>> >> >> > records.
>> >> >> > 11/09/16 07:58:07 ERROR tool.ImportTool: Error during import:
>> >> >> > Import
>> >> >> > job failed!
>> >> >> >
>> >> >> > --
>> >> >> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated
in
>> >> >> > favor
>> >> >> > of
>> >> >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org.
Please
>> >> >> > subscribe
>> >> >> > to it by sending an email to
>> >> >> > incubator-sqoop-user-subscribe@apache.org.
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in
>> >> >> favor
>> >> >> of
>> >> >> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> >> >> subscribe
>> >> >> to it by sending an email to
>> >> >> incubator-sqoop-user-subscribe@apache.org.
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Eric H.
>> >> > eric.hardway@gmail.com
>> >> >
>> >> > --
>> >> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor
>> >> > of
>> >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> >> > subscribe
>> >> > to it by sending an email to
>> >> > incubator-sqoop-user-subscribe@apache.org.
>> >> >
>> >>
>> >> --
>> >> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor
>> >> of
>> >> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> >> subscribe
>> >> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>> >
>> >
>> >
>> > --
>> > Eric H.
>> > eric.hardway@gmail.com
>> >
>> > --
>> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
>> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> > subscribe
>> > to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>> >
>>
>> --
>> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
>> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe
>> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>
>
>
> --
> Eric H.
> eric.hardway@gmail.com
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe
> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>

Mime
View raw message