sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle B <kbi...@gmail.com>
Subject Sqoop 2 Import
Date Fri, 03 May 2013 19:11:05 GMT
Hello,

I am migrating from Sqoop 1.4.3 to Sqoop 2, and am kind of stumbling my way
through the intro stuff. I'd like to import data from a MySQL database into
HDFS, and a simple import seems to be failing in 2, which works fine in 1.

- From (1.4.3) -
sqoop import --connect jdbc:mysql://127.0.0.1:3306/db --username kyle -P
--table kyle_table --target-dir /user/kyle/table

- To (1.99.2) -
sqoop:000> create job --xid 1 --type import
Creating job for connection with id 1
Please fill following values to create new job object
Name: test

Database configuration

Schema name:
Table name: kyle_table
Table SQL statement:
Table column names: *
Partition column name:
Boundary query:

Output configuration

Storage type:
  0 : HDFS
Choose: 0
Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 0
Output directory: /user/kyle/table2

Throttling resources

Extractors:
Loaders:
New job was successfully created with validation status FINE  and
persistent id 12
sqoop:000> submission start --jid 12
Submission details
Job id: 12
Status: BOOTING
Creation date: 2013-05-03 11:38:01 MST
Last update date: 2013-05-03 11:38:01 MST
External Id: job_1367275490217_0075
        http://server:8088/proxy/application_1367275490217_0075/
Progress: Progress is not available
sqoop:000> submission status --jid 12
Exception has occurred during processing command
Server has returned exception: Exception: java.lang.NullPointerException
Message:


--
On both cases, I see the jobs made their way to the job history, and on the
first, the output was saved to HDFS. On the second, I just have a blank
/user/kyle/table2 folder, and the task failed in Hadoop.


 - Hadoop Logs -
 2013-05-03 11:38:10,172 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.NullPointerException
 at java.lang.String.<init>(Unknown Source)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38)
 at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Unknown Source)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)

2013-05-03 11:38:10,172 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while loading
data out of MR job.
java.lang.NullPointerException
 at java.lang.String.<init>(Unknown Source)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.getFrameworkConnection(ConfigurationUtils.java:50)
 at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:204)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)


- sqoop.log -
2013-05-03 11:38:01,439 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:01,452 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2013-05-03 11:38:01,453 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:01,464 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2013-05-03 11:38:01,465 DEBUG framework.FrameworkManager
[org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:417)]
Using callbacks:
Importer{initializer=org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer,
destroyer=org.apache.sqoop.connector.jdbc.GenericJdbcImportDestroyer,
partitioner=org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner,
extractor=org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor}
2013-05-03 11:38:01,482 DEBUG jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:154)]
Using minMaxQuery: SELECT MIN(id), MAX(id) FROM kyle_table
2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:169)]
Boundaries: min=1, max=4, columnType=4
2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:263)]
Using dataSql: SELECT * FROM kyle_table WHERE ${CONDITIONS}
2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:264)]
Field names: *
2013-05-03 11:38:01,483 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:01,485 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2013-05-03 11:38:01,485 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.2.jar
2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-core-1.99.2.jar
2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-spi-1.99.2.jar
2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-execution-mapreduce-1.99.2-hadoop200.jar
2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2.jar
2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/json-simple-1.1.jar
2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/mysql-connector-java-5.1.18-bin.jar
2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
Adding jar to the job:
file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/guava-11.0.2.jar
2013-05-03 11:38:01,519 WARN  mapreduce.JobSubmitter
[org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:138)]
Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
2013-05-03 11:38:01,913 WARN  mapreduce.JobSubmitter
[org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:247)]
No job jar file set.  User classes may not be found. See Job or
Job#setJar(String).
2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat
[org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
Partition: 1 <= id AND id < 2
2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat
[org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
Partition: 2 <= id AND id < 3
2013-05-03 11:38:01,945 DEBUG mr.SqoopInputFormat
[org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
Partition: 3 <= id AND id <= 4
2013-05-03 11:38:02,010 WARN  conf.Configuration
[org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)]
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
2013-05-03 11:38:02,010 WARN  conf.Configuration
[org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)]
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
2013-05-03 11:38:02,126 DEBUG mapreduce.MapreduceSubmissionEngine
[org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:272)]
Executed new map-reduce job with id job_1367275490217_0075
2013-05-03 11:38:02,126 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:02,129 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit



--
Can anyone point out anything i'm missing? I'm following the quick start
for creating a connection and first job. I would have thought the above
would have dumped kyle_table to hdfs:/user/kyle/table2 . Am I using the
right method to import a table to HDFS using Sqoop 2?

Thanks,

-Kyle

Mime
View raw message