sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Sqoop 2 Import
Date Sat, 04 May 2013 00:40:49 GMT
Hi Keyle,
I'm having issues to find that precise version. It do not seems to be available on Apache
archive [1] and I did not find any branch or tag related to it on github mirror [2]. Would
you mind sharing with us where did you downloaded it?

Jarcec

Links:
1: http://archive.apache.org/dist/hadoop/common/
2: https://github.com/apache/hadoop-common

On Fri, May 03, 2013 at 02:52:00PM -0700, Kyle B wrote:
> Hi Jarcec,
> 
> I'm currently running Hadoop 2.0.2.1-alpha. So it should have that fix in
> theory.
> 
> -Kyle
> 
> On Fri, May 3, 2013 at 2:41 PM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:
> 
> > Hi Kyle,
> > what Hadoop version are you using?
> >
> > Based on the exception it seems that you're using YARN which is suggesting
> > Hadoop 2. I hit similar NullPointerException during release candidate
> > testing on Hadoop 2.0.0-alpha. There is a known HDFS bug HADOOP-8726 [1]
> > that is causing this. It was fixed in 2.0.2-alpha, so I'm wondering if
> > you're using 2.0.0 or 2.0.1 by any chance?
> >
> > Jarcec
> >
> > Links:
> > 1: https://issues.apache.org/jira/browse/HADOOP-8726
> >
> > On Fri, May 03, 2013 at 12:11:05PM -0700, Kyle B wrote:
> > > Hello,
> > >
> > > I am migrating from Sqoop 1.4.3 to Sqoop 2, and am kind of stumbling my
> > way
> > > through the intro stuff. I'd like to import data from a MySQL database
> > into
> > > HDFS, and a simple import seems to be failing in 2, which works fine in
> > 1.
> > >
> > > - From (1.4.3) -
> > > sqoop import --connect jdbc:mysql://127.0.0.1:3306/db --username kyle -P
> > > --table kyle_table --target-dir /user/kyle/table
> > >
> > > - To (1.99.2) -
> > > sqoop:000> create job --xid 1 --type import
> > > Creating job for connection with id 1
> > > Please fill following values to create new job object
> > > Name: test
> > >
> > > Database configuration
> > >
> > > Schema name:
> > > Table name: kyle_table
> > > Table SQL statement:
> > > Table column names: *
> > > Partition column name:
> > > Boundary query:
> > >
> > > Output configuration
> > >
> > > Storage type:
> > >   0 : HDFS
> > > Choose: 0
> > > Output format:
> > >   0 : TEXT_FILE
> > >   1 : SEQUENCE_FILE
> > > Choose: 0
> > > Output directory: /user/kyle/table2
> > >
> > > Throttling resources
> > >
> > > Extractors:
> > > Loaders:
> > > New job was successfully created with validation status FINE  and
> > > persistent id 12
> > > sqoop:000> submission start --jid 12
> > > Submission details
> > > Job id: 12
> > > Status: BOOTING
> > > Creation date: 2013-05-03 11:38:01 MST
> > > Last update date: 2013-05-03 11:38:01 MST
> > > External Id: job_1367275490217_0075
> > >         http://server:8088/proxy/application_1367275490217_0075/
> > > Progress: Progress is not available
> > > sqoop:000> submission status --jid 12
> > > Exception has occurred during processing command
> > > Server has returned exception: Exception: java.lang.NullPointerException
> > > Message:
> > >
> > >
> > > --
> > > On both cases, I see the jobs made their way to the job history, and on
> > the
> > > first, the output was saved to HDFS. On the second, I just have a blank
> > > /user/kyle/table2 folder, and the task failed in Hadoop.
> > >
> > >
> > >  - Hadoop Logs -
> > >  2013-05-03 11:38:10,172 WARN [main] org.apache.hadoop.mapred.YarnChild:
> > > Exception running child : java.lang.NullPointerException
> > >  at java.lang.String.<init>(Unknown Source)
> > >  at
> > >
> > org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
> > >  at
> > >
> > org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38)
> > >  at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69)
> > >  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
> > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
> > >  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> > >  at java.security.AccessController.doPrivileged(Native Method)
> > >  at javax.security.auth.Subject.doAs(Unknown Source)
> > >  at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> > >  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> > >
> > > 2013-05-03 11:38:10,172 ERROR [OutputFormatLoader-consumer]
> > > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while
> > loading
> > > data out of MR job.
> > > java.lang.NullPointerException
> > >  at java.lang.String.<init>(Unknown Source)
> > >  at
> > >
> > org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
> > >  at
> > >
> > org.apache.sqoop.job.mr.ConfigurationUtils.getFrameworkConnection(ConfigurationUtils.java:50)
> > >  at
> > >
> > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:204)
> > >  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> > >  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> > >  at java.util.concurrent.FutureTask.run(Unknown Source)
> > >  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> > Source)
> > >  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > >  at java.lang.Thread.run(Unknown Source)
> > >
> > >
> > > - sqoop.log -
> > > 2013-05-03 11:38:01,439 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
> > > Tx count-begin: 1, rollback: false
> > > 2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
> > > Tx count-commit: 1, rollback: false
> > > 2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
> > > Tx count-close: 0, rollback: false
> > > 2013-05-03 11:38:01,452 INFO  repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
> > > Attempting transaction commit
> > > 2013-05-03 11:38:01,453 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
> > > Tx count-begin: 1, rollback: false
> > > 2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
> > > Tx count-commit: 1, rollback: false
> > > 2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
> > > Tx count-close: 0, rollback: false
> > > 2013-05-03 11:38:01,464 INFO  repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
> > > Attempting transaction commit
> > > 2013-05-03 11:38:01,465 DEBUG framework.FrameworkManager
> > >
> > [org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:417)]
> > > Using callbacks:
> > >
> > Importer{initializer=org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer,
> > > destroyer=org.apache.sqoop.connector.jdbc.GenericJdbcImportDestroyer,
> > > partitioner=org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner,
> > > extractor=org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor}
> > > 2013-05-03 11:38:01,482 DEBUG jdbc.GenericJdbcImportInitializer
> > >
> > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:154)]
> > > Using minMaxQuery: SELECT MIN(id), MAX(id) FROM kyle_table
> > > 2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
> > >
> > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:169)]
> > > Boundaries: min=1, max=4, columnType=4
> > > 2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
> > >
> > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:263)]
> > > Using dataSql: SELECT * FROM kyle_table WHERE ${CONDITIONS}
> > > 2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
> > >
> > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:264)]
> > > Field names: *
> > > 2013-05-03 11:38:01,483 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
> > > Tx count-begin: 1, rollback: false
> > > 2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
> > > Tx count-commit: 1, rollback: false
> > > 2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
> > > Tx count-close: 0, rollback: false
> > > 2013-05-03 11:38:01,485 INFO  repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
> > > Attempting transaction commit
> > > 2013-05-03 11:38:01,485 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.2.jar
> > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-core-1.99.2.jar
> > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-spi-1.99.2.jar
> > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-execution-mapreduce-1.99.2-hadoop200.jar
> > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2.jar
> > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/json-simple-1.1.jar
> > > 2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/mysql-connector-java-5.1.18-bin.jar
> > > 2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)]
> > > Adding jar to the job:
> > >
> > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/guava-11.0.2.jar
> > > 2013-05-03 11:38:01,519 WARN  mapreduce.JobSubmitter
> > >
> > [org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:138)]
> > > Hadoop command-line option parsing not performed. Implement the Tool
> > > interface and execute your application with ToolRunner to remedy this.
> > > 2013-05-03 11:38:01,913 WARN  mapreduce.JobSubmitter
> > >
> > [org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:247)]
> > > No job jar file set.  User classes may not be found. See Job or
> > > Job#setJar(String).
> > > 2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat
> > >
> > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
> > > Partition: 1 <= id AND id < 2
> > > 2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat
> > >
> > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
> > > Partition: 2 <= id AND id < 3
> > > 2013-05-03 11:38:01,945 DEBUG mr.SqoopInputFormat
> > >
> > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)]
> > > Partition: 3 <= id AND id <= 4
> > > 2013-05-03 11:38:02,010 WARN  conf.Configuration
> > >
> > [org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)]
> > > mapred.map.tasks.speculative.execution is deprecated. Instead, use
> > > mapreduce.map.speculative
> > > 2013-05-03 11:38:02,010 WARN  conf.Configuration
> > >
> > [org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)]
> > > mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> > > mapreduce.reduce.speculative
> > > 2013-05-03 11:38:02,126 DEBUG mapreduce.MapreduceSubmissionEngine
> > >
> > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:272)]
> > > Executed new map-reduce job with id job_1367275490217_0075
> > > 2013-05-03 11:38:02,126 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
> > > Tx count-begin: 1, rollback: false
> > > 2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
> > > Tx count-commit: 1, rollback: false
> > > 2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
> > > Tx count-close: 0, rollback: false
> > > 2013-05-03 11:38:02,129 INFO  repository.JdbcRepositoryTransaction
> > >
> > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
> > > Attempting transaction commit
> > >
> > >
> > >
> > > --
> > > Can anyone point out anything i'm missing? I'm following the quick start
> > > for creating a connection and first job. I would have thought the above
> > > would have dumped kyle_table to hdfs:/user/kyle/table2 . Am I using the
> > > right method to import a table to HDFS using Sqoop 2?
> > >
> > > Thanks,
> > >
> > > -Kyle
> >

Mime
View raw message