sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kate Ting <k...@cloudera.com>
Subject Re: [sqoop-user] Problem using sqoop with --direct (mysqldump)
Date Sat, 17 Sep 2011 18:43:53 GMT
Eric, what is your mysql version and your mysqldump version?

Regards, Kate

On Fri, Sep 16, 2011 at 7:30 PM, eric hernandez <eric.hardway@gmail.com> wrote:
> Kate,
>
> sqoop import --connect jdbc:mysql://192.168.0.100:3307/test --verbose -m 1
> --username sqoop --password sanitized --hive-overwrite --direct --table
> table1 --hive-import --create-hive-table --hive-table table1
> --fields-terminated-by '\t' --lines-terminated-by '\n' --append
>
> Please note this works fine if i remove the --direct option. I am also
> limiting it to 1 mapper because if not the output of the failure is very
> long.
>
>
> On Fri, Sep 16, 2011 at 3:01 PM, Kate Ting <kate@cloudera.com> wrote:
>>
>> Eric - what is the exact Sqoop command that you ran (including, if
>> applicable, contents of the options-file)?
>>
>> Regards, Kate
>>
>> On Fri, Sep 16, 2011 at 9:27 AM, eric hernandez <eric.hardway@gmail.com>
>> wrote:
>> > Yes i have mysqldump on all nodes.
>> >
>> > Verbose output
>> >
>> >
>> >
>> > 11/09/16 09:22:42 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>> > 11/09/16 09:22:42 WARN tool.BaseSqoopTool: Setting your password on the
>> > command-line is insecure. Consider using -P instead.
>> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Loaded manager factory:
>> > com.cloudera.sqoop.manager.DefaultManagerFactory
>> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>> > com.cloudera.sqoop.manager.DefaultManagerFactory
>> > 11/09/16 09:22:42 DEBUG manager.DefaultManagerFactory: Trying with
>> > scheme:
>> > jdbc:mysql:
>> > 11/09/16 09:22:42 INFO manager.MySQLManager: Preparing to use a MySQL
>> > streaming resultset.
>> > 11/09/16 09:22:42 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>> > com.cloudera.sqoop.manager.DirectMySQLManager@7ad81784
>> > 11/09/16 09:22:42 INFO tool.CodeGenTool: Beginning code generation
>> > 11/09/16 09:22:42 DEBUG manager.SqlManager: No connection paramenters
>> > specified. Using regular API for making connection.
>> > 11/09/16 09:22:43 DEBUG manager.SqlManager: Using fetchSize for next
>> > query:
>> > -2147483648
>> > 11/09/16 09:22:43 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT
>> > t.* FROM `table1` AS t LIMIT 1
>> > 11/09/16 09:22:43 DEBUG manager.SqlManager: Using fetchSize for next
>> > query:
>> > -2147483648
>> > 11/09/16 09:22:43 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT
>> > t.* FROM `table1` AS t LIMIT 1
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: selected columns:
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   id
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   application_id
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   event_id
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   response_id
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   target_id
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   mode
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter:   date_created
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Writing source file:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.java
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Table name: table1
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: Columns: id:4,
>> > application_id:4,
>> > event_id:4, response_id:4, target_id:4, mode:1, date_created:93,
>> > 11/09/16 09:22:43 DEBUG orm.ClassWriter: sourceFilename is table1.java
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Found existing
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> > 11/09/16 09:22:43 INFO orm.CompilationManager: HADOOP_HOME is
>> > /usr/lib/hadoop
>> > 11/09/16 09:22:43 INFO orm.CompilationManager: Found hadoop core jar at:
>> > /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Adding source file:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.java
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager: Invoking javac with
>> > args:
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -sourcepath
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -d
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:   -classpath
>> > 11/09/16 09:22:43 DEBUG orm.CompilationManager:
>> >
>> > /etc/hadoop/conf:/usr/java/jdk1.6.0_21/lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/hadoop-fairscheduler-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/lib/hue-plugins-1.2.0-cdh3u1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/junit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.2.2.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/sqoop/conf:/etc/zookeeper::/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar:/usr/lib/sqoop/lib/avro-1.5.1.jar:/usr/lib/sqoop/lib/avro-ipc-1.5.1.jar:/usr/lib/sqoop/lib/avro-mapred-1.5.1.jar:/usr/lib/sqoop/lib/commons-io-1.4.jar:/usr/lib/sqoop/lib/ivy-2.0.0-rc2.jar:/usr/lib/sqoop/lib/jackson-core-asl-1.7.3.jar:/usr/lib/sqoop/lib/jackson-mapper-asl-1.7.3.jar:/usr/lib/sqoop/lib/jopt-simple-3.2.jar:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar:/usr/lib/sqoop/lib/paranamer-2.3.jar:/usr/lib/sqoop/lib/snappy-java-1.0.3-rc2.jar:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar:/usr/lib/sqoop/sqoop-test-1.3.0-cdh3u1.jar::/usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> > 11/09/16 09:22:44 INFO orm.CompilationManager: Writing jar file:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.jar
>> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Scanning for .class
>> > files in
>> > directory: /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b
>> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Got classfile:
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.class ->
>> > table1.class
>> > 11/09/16 09:22:44 DEBUG orm.CompilationManager: Finished writing jar
>> > file
>> > /tmp/sqoop-root/compile/4da62fc9c254b3faa3ba5115ef61783b/table1.jar
>> > 11/09/16 09:22:44 DEBUG tool.ImportTool: Using temporary folder:
>> > 16092244516table1
>> > 11/09/16 09:22:44 INFO manager.DirectMySQLManager: Beginning mysqldump
>> > fast
>> > path import
>> > 11/09/16 09:22:44 INFO mapreduce.ImportJobBase: Beginning import of
>> > table1
>> > 11/09/16 09:22:44 DEBUG manager.SqlManager: Using fetchSize for next
>> > query:
>> > -2147483648
>> > 11/09/16 09:22:44 INFO manager.SqlManager: Executing SQL statement:
>> > SELECT
>> > t.* FROM `table1` AS t LIMIT 1
>> > 11/09/16 09:22:44 DEBUG mapreduce.MySQLDumpImportJob: Using InputFormat:
>> > class com.cloudera.sqoop.mapreduce.MySQLDumpInputFormat
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/sqoop-1.3.0-cdh3u1.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/jackson-mapper-asl-1.7.3.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/snappy-java-1.0.3-rc2.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.15-bin.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/paranamer-2.3.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/ivy-2.0.0-rc2.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/jackson-core-asl-1.7.3.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/avro-mapred-1.5.1.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/avro-1.5.1.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/jopt-simple-3.2.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/commons-io-1.4.jar
>> > 11/09/16 09:22:44 DEBUG mapreduce.JobBase: Adding to job classpath:
>> > file:/usr/lib/sqoop/lib/avro-ipc-1.5.1.jar
>> > 11/09/16 09:22:46 INFO mapred.JobClient: Running job:
>> > job_201109160744_0006
>> > 11/09/16 09:22:47 INFO mapred.JobClient:  map 0% reduce 0%
>> > 11/09/16 09:22:53 INFO mapred.JobClient: Task Id :
>> > attempt_201109160744_0006_m_000000_0, Status : FAILED
>> > java.io.IOException: mysqldump terminated with status 5
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >     at java.security.AccessController.doPrivileged(Native Method)
>> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >     at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >
>> > attempt_201109160744_0006_m_000000_0: Exception in thread "Thread-12"
>> > java.lang.IndexOutOfBoundsException
>> > attempt_201109160744_0006_m_000000_0:     at
>> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> > attempt_201109160744_0006_m_000000_0:     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> > attempt_201109160744_0006_m_000000_0: log4j:WARN No appenders could be
>> > found
>> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> > attempt_201109160744_0006_m_000000_0: log4j:WARN Please initialize the
>> > log4j
>> > system properly.
>> > 11/09/16 09:22:58 INFO mapred.JobClient: Task Id :
>> > attempt_201109160744_0006_m_000000_1, Status : FAILED
>> > java.io.IOException: mysqldump terminated with status 5
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >     at java.security.AccessController.doPrivileged(Native Method)
>> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >     at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >
>> > attempt_201109160744_0006_m_000000_1: Exception in thread "Thread-12"
>> > java.lang.IndexOutOfBoundsException
>> > attempt_201109160744_0006_m_000000_1:     at
>> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> > attempt_201109160744_0006_m_000000_1:     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> > attempt_201109160744_0006_m_000000_1: log4j:WARN No appenders could be
>> > found
>> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> > attempt_201109160744_0006_m_000000_1: log4j:WARN Please initialize the
>> > log4j
>> > system properly.
>> > 11/09/16 09:23:03 INFO mapred.JobClient: Task Id :
>> > attempt_201109160744_0006_m_000000_2, Status : FAILED
>> > java.io.IOException: mysqldump terminated with status 5
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >     at java.security.AccessController.doPrivileged(Native Method)
>> >     at javax.security.auth.Subject.doAs(Subject.java:396)
>> >     at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >     at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >
>> > attempt_201109160744_0006_m_000000_2: Exception in thread "Thread-12"
>> > java.lang.IndexOutOfBoundsException
>> > attempt_201109160744_0006_m_000000_2:     at
>> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> > attempt_201109160744_0006_m_000000_2:     at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> > attempt_201109160744_0006_m_000000_2: log4j:WARN No appenders could be
>> > found
>> > for logger (org.apache.hadoop.hdfs.DFSClient).
>> > attempt_201109160744_0006_m_000000_2: log4j:WARN Please initialize the
>> > log4j
>> > system properly.
>> > 11/09/16 09:23:09 INFO mapred.JobClient: Job complete:
>> > job_201109160744_0006
>> > 11/09/16 09:23:09 INFO mapred.JobClient: Counters: 6
>> > 11/09/16 09:23:09 INFO mapred.JobClient:   Job Counters
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=19196
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     Total time spent by all
>> > reduces
>> > waiting after reserving slots (ms)=0
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     Total time spent by all
>> > maps
>> > waiting after reserving slots (ms)=0
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     Launched map tasks=4
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
>> > 11/09/16 09:23:09 INFO mapred.JobClient:     Failed map tasks=1
>> > 11/09/16 09:23:09 INFO mapreduce.ImportJobBase: Transferred 0 bytes in
>> > 24.8354 seconds (0 bytes/sec)
>> > 11/09/16 09:23:09 INFO mapreduce.ImportJobBase: Retrieved 0 records.
>> > 11/09/16 09:23:09 ERROR tool.ImportTool: Error during import: Import job
>> > failed!
>> >
>> >
>> > --- Task log
>> >
>> > Task Logs: 'attempt_201109160744_0006_m_000000_1'
>> >
>> > stdout logs
>> > ________________________________
>> >
>> > stderr logs
>> >
>> > Exception in thread "Thread-12" java.lang.
>> > IndexOutOfBoundsException
>> >       at java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >       at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink$ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> > log4j:WARN No appenders could be found for logger
>> > (org.apache.hadoop.hdfs.DFSClient).
>> > log4j:WARN Please initialize the log4j system properly.
>> >
>> > ________________________________
>> >
>> > syslog logs
>> >
>> > 2011-09-16 09:22:54,194 WARN org.apache.hadoop.util.NativeCodeLoader:
>> > Unable
>> > to load native-hadoop library for your platform... using builtin-java
>> > classes where applicable
>> > 2011-09-16 09:22:54,326 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> > Initializing JVM Metrics with processName=MAP, sessionId=
>> > 2011-09-16 09:22:54,687 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > Beginning mysqldump fast path import
>> > 2011-09-16 09:22:54,690 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > Performing import of table table1 from database test
>> > 2011-09-16 09:22:54,696 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > Converting data to use specified delimiters.
>> > 2011-09-16 09:22:54,696 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > (For the fastest possible import, use
>> > 2011-09-16 09:22:54,696 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > --mysql-delimiters to specify the same field
>> > 2011-09-16 09:22:54,696 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > delimiters as are used by mysqldump.)
>> > 2011-09-16 09:22:54,710 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > mysqldump: Got errno 32 on write
>> > 2011-09-16 09:22:54,710 INFO
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper:
>> > Transfer loop complete.
>> > 2011-09-16 09:22:54,740 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> > Initializing logs' truncater with mapRetainSize=-1 and
>> > reduceRetainSize=-1
>> > 2011-09-16 09:22:54,746 WARN org.apache.hadoop.mapred.Child: Error
>> > running
>> > child
>> > java.io.IOException: mysqldump terminated with status 5
>> >       at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:476)
>> >       at
>> >
>> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
>> >       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >       at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at javax.security.auth.Subject.doAs(Subject.java:396)
>> >       at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>> >       at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> > 2011-09-16 09:22:54,750 INFO org.apache.hadoop.mapred.Task: Runnning
>> > cleanup
>> > for the task
>> >
>> >
>> > On Fri, Sep 16, 2011 at 8:55 AM, Kate Ting <kate@cloudera.com> wrote:
>> >>
>> >> [Moving the conversation to sqoop-user@incubator.apache.org. Please
>> >> subscribe (and post questions) to the new mailing list.]
>> >>
>> >> Hi Eric -
>> >>
>> >> (1) Is the mysqldump utility installed on individual node machines?
>> >> (2) If so, can you pastebin your task log as well as verbose output?
>> >>
>> >> Regards, Kate
>> >>
>> >> On Fri, Sep 16, 2011 at 8:04 AM, Eric <eric.hardway@gmail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > I cannot sqoop in using the --direct option, my sqoop works fine up
>> >> > until i add --direct .
>> >> >
>> >> > I am using Sqoop 1.3.0-cdh3u1
>> >> > git commit id 3a60cc809b14d538dd1eb0e90ffa9767e8d06a43
>> >> > Compiled by jenkins@ubuntu-slave01 on Mon Jul 18 08:38:49 PDT 2011
>> >> >
>> >> > Please Advise,
>> >> >
>> >> > -Eric
>> >> >
>> >> >
>> >> > error message:
>> >> >
>> >> > 11/09/16 07:57:39 INFO manager.MySQLManager: Preparing to use a MySQL
>> >> > streaming resultset.
>> >> > 11/09/16 07:57:39 INFO tool.CodeGenTool: Beginning code generation
>> >> > 11/09/16 07:57:40 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 07:57:40 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 07:57:40 INFO orm.CompilationManager: HADOOP_HOME is
>> >> > /usr/lib/
>> >> > hadoop
>> >> > 11/09/16 07:57:40 INFO orm.CompilationManager: Found hadoop core jar
>> >> > at: /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar
>> >> > 11/09/16 07:57:41 INFO orm.CompilationManager: Writing jar file:
>> >> > /tmp/
>> >> > sqoop-root/compile/aef5c62d2156aeae5338ee272de42d26/table1.jar
>> >> > 11/09/16 07:57:41 INFO manager.DirectMySQLManager: Beginning
>> >> > mysqldump
>> >> > fast path import
>> >> > 11/09/16 07:57:41 INFO mapreduce.ImportJobBase: Beginning import of
>> >> > table1
>> >> > 11/09/16 07:57:41 INFO manager.SqlManager: Executing SQL statement:
>> >> > SELECT t.* FROM `table1` AS t LIMIT 1
>> >> > 11/09/16 07:57:43 INFO mapred.JobClient: Running job:
>> >> > job_201109160744_0004
>> >> > 11/09/16 07:57:44 INFO mapred.JobClient:  map 0% reduce 0%
>> >> > 11/09/16 07:57:50 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0004_m_000000_0, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 476)
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 49)
>> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >        at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >        at java.security.AccessController.doPrivileged(Native Method)
>> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >        at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> > 1127)
>> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0004_m_000000_0: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0004_m_000000_0:   at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0004_m_000000_0:   at
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0004_m_000000_0: log4j:WARN No appenders could
>> >> > be
>> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0004_m_000000_0: log4j:WARN Please initialize
>> >> > the
>> >> > log4j system properly.
>> >> > 11/09/16 07:57:55 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0004_m_000000_1, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 476)
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 49)
>> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >        at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >        at java.security.AccessController.doPrivileged(Native Method)
>> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >        at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> > 1127)
>> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0004_m_000000_1: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0004_m_000000_1:   at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0004_m_000000_1:   at
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0004_m_000000_1: log4j:WARN No appenders could
>> >> > be
>> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0004_m_000000_1: log4j:WARN Please initialize
>> >> > the
>> >> > log4j system properly.
>> >> > 11/09/16 07:58:01 INFO mapred.JobClient: Task Id :
>> >> > attempt_201109160744_0004_m_000000_2, Status : FAILED
>> >> > java.io.IOException: mysqldump terminated with status 5
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 476)
>> >> >        at
>> >> >
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:
>> >> > 49)
>> >> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> >> >        at
>> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >> >        at java.security.AccessController.doPrivileged(Native Method)
>> >> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >> >        at
>> >> >
>> >> >
>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
>> >> > 1127)
>> >> >        at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >> >
>> >> > attempt_201109160744_0004_m_000000_2: Exception in thread "Thread-12"
>> >> > java.lang.IndexOutOfBoundsException
>> >> > attempt_201109160744_0004_m_000000_2:   at
>> >> > java.nio.CharBuffer.wrap(CharBuffer.java:445)
>> >> > attempt_201109160744_0004_m_000000_2:   at
>> >> > com.cloudera.sqoop.mapreduce.MySQLDumpMapper$ReparsingAsyncSink
>> >> > $ReparsingStreamThread.run(MySQLDumpMapper.java:253)
>> >> > attempt_201109160744_0004_m_000000_2: log4j:WARN No appenders could
>> >> > be
>> >> > found for logger (org.apache.hadoop.hdfs.DFSClient).
>> >> > attempt_201109160744_0004_m_000000_2: log4j:WARN Please initialize
>> >> > the
>> >> > log4j system properly.
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient: Job complete:
>> >> > job_201109160744_0004
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient: Counters: 6
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:   Job Counters
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=19165
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Total time spent by
all
>> >> > reduces waiting after reserving slots (ms)=0
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Total time spent by
all
>> >> > maps waiting after reserving slots (ms)=0
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Launched map tasks=4
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
>> >> > 11/09/16 07:58:07 INFO mapred.JobClient:     Failed map tasks=1
>> >> > 11/09/16 07:58:07 INFO mapreduce.ImportJobBase: Transferred 0 bytes
>> >> > in
>> >> > 25.1844 seconds (0 bytes/sec)
>> >> > 11/09/16 07:58:07 INFO mapreduce.ImportJobBase: Retrieved 0 records.
>> >> > 11/09/16 07:58:07 ERROR tool.ImportTool: Error during import: Import
>> >> > job failed!
>> >> >
>> >> > --
>> >> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor
>> >> > of
>> >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> >> > subscribe
>> >> > to it by sending an email to
>> >> > incubator-sqoop-user-subscribe@apache.org.
>> >> >
>> >>
>> >> --
>> >> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor
>> >> of
>> >> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> >> subscribe
>> >> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>> >
>> >
>> >
>> > --
>> > Eric H.
>> > eric.hardway@gmail.com
>> >
>> > --
>> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
>> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>> > subscribe
>> > to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>> >
>>
>> --
>> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
>> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe
>> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>
>
>
> --
> Eric H.
> eric.hardway@gmail.com
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of
> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe
> to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>

Mime
View raw message