sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Millstein <joshua.millst...@gmail.com>
Subject Re: import mysql query to hbase using sqoop
Date Thu, 05 Jun 2014 19:01:00 GMT
I can not import directly into hdfs either.

Command : sqoop import --connect
jdbc:mysql://wolfs-MacBook-Pro.local:3306/joshLocal --username sqoop
--password sqoop --table BITLOG


Error:

 jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker,
sessionId=
14/06/05 13:56:11 INFO mapreduce.JobSubmitter: Cleaning up the staging area
file:/tmp/hadoop-wolf/mapred/staging/wolf957353874/.staging/job_local957353874_0001
14/06/05 13:56:11 ERROR tool.ImportTool: Encountered IOException running
import job: java.io.FileNotFoundException: File does not exist:
hdfs://wolfs-MacBook-Pro.local:8020/usr/local/Cellar/sqoop/1.4.4/libexec/lib/jopt-simple-3.2.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

---ALSO

output from find is attached.  I see a lot of hadoop 2.2 references whereas
I am running 2.4, problem there ?


On Thu, Jun 5, 2014 at 10:01 AM, Josh Millstein <joshua.millstein@gmail.com>
wrote:

> Hi Abe - Thanks for the response.
>
> I've updated my sqoop-env.sh file to have the following export vars
>
> export HADOOP_HOME="/usr/local/Cellar/hadoop/2.4.0/libexec"
> export HBASE_HOME="/usr/local/Cellar/hbase/0.98.1/libexec"
> export HIVE_HOME="/usr/local/Cellar/hive/0.13.0/libexec"
> export ZOOCFGDIR="/usr/local/etc/zookeeper"
> export HCAT_HOME="/usr/local/Cellar/hive/0.13.0/libexec/hcatalog"
>
> and to answer your question the hbase directory has the following file /
> directories
>
> CHANGES.txt LICENSE.txt README.txt libexec
> INSTALL_RECEIPT.json NOTICE.txt bin
>
> Now, when I try and run my import I get a new error (and the hbase table
> is actually created, just no data going into it)
>
> The error is
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
> at
> org.apache.sqoop.mapreduce.DelegatingOutputFormat.checkOutputSpecs(DelegatingOutputFormat.java:63)
>  at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
>  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
>  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
>  at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
>  at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> a
>
> I've read this is because of incompatible jar files but, like I said I'm a
> complete newb at all of this and if I could get a little more direction on
> what exactly I'm missing / is incompatible that would be fantastic.
>
> This is all on homebrew mac os x 10.9 hadoop 2.4 hbase .98 sqoop 1.4.4
>
>
>
>
> On Thu, Jun 5, 2014 at 2:14 AM, Abraham Elmahrek <abe@cloudera.com> wrote:
>
>> Hey there,
>>
>> What does the contents of the directory '/usr/local/Cellar/hbase/0.98.1'
>> look like? The command 'find /usr/local/Cellar/hbase/0.98.1' should
>> assist in figuring that out. Please paste the result of that command here.
>>
>> The environment variables listed above are used in many different areas
>> of Hadoop. I've typically seen them in startup scripts to resolve libraries
>> and binaries. In this case, it's used to include jars in the classpath when
>> you call Sqoop.
>>
>> Are you able to import to HDFS?
>>
>> -Abe
>>
>>
>> On Wed, Jun 4, 2014 at 6:59 PM, Josh Millstein <
>> joshua.millstein@gmail.com> wrote:
>>
>>> I've got hadoop / hbase / hive / set up and running (can create files on
>>> hdfs, run map reduce jobs, create a "table" in hbase and also in hive) on
>>> my mac with osx 10.9. I'm now trying to import data from a mysql table into
>>> sqoop (using query, not table-name etc). I am getting this error with this
>>> command.
>>>
>>> --COMMAND
>>> sqoop import --connect jdbc:mysql:///joshLocal --username root --query "SELECT
* FROM     BITLOG WHERE \$CONDITIONS" --split-by oozie_job.id --hbase-table bitlogTest --hbase-create-
   table --column-family bitLogColumn
>>> --ERROR
>>> ERROR tool.ImportTool: Error during import: HBase jars are not present in classpath,
cannot import to HBase!
>>>
>>>  I believe that all the export vars are correctly setup. I have the
>>> following in swoop-env.sh
>>>
>>> export HADOOP_HOME="/usr/local/Cellar/hadoop/2.4.0"
>>> export HBASE_HOME="/usr/local/Cellar/hbase/0.98.1"
>>> export HIVE_HOME="/usr/local/Cellar/hive/0.13.0"
>>> export ZOOCFGDIR="/usr/local/etc/zookeeper"
>>> export HCAT_HOME="/usr/local/Cellar/hive/0.13.0/libexec/hcatalog"
>>>
>>>  one thing I did that gave a different message was to change the hbase
>>> home to point to HBASE_HOME/libexec/lib in swoop-env.sh. That gave me a
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/Tool
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>
>>>  error. I have seen some advice given that says I need to copy over
>>> hadoop security jar files to hbase's installation. I don't exactly know
>>> what files need to go over and if that is even an issue. The only reason I
>>> thought it might be is because java.security.SecureClassLoader is in the
>>> stacktrace.
>>>
>>> I'm sorry if this is a really basic java question but I'm a complete
>>> novice with it.
>>>
>>> One other even more basic java question. When we define HADOOP_HOME
>>> HBASE_HOME etc., what are we "telling" the other java programs that rely on
>>> that info. Are we saying "here is the executable java file" or are we
>>> saying "here are the jar file in the lib folder" . I don't quite understand
>>> what I should actually be pointing to because I don't know how that path is
>>> used.
>>>
>>
>>
>

Mime
View raw message