sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Millstein <joshua.millst...@gmail.com>
Subject Re: import mysql query to hbase using sqoop
Date Thu, 05 Jun 2014 15:01:27 GMT
Hi Abe - Thanks for the response.

I've updated my sqoop-env.sh file to have the following export vars

export HADOOP_HOME="/usr/local/Cellar/hadoop/2.4.0/libexec"
export HBASE_HOME="/usr/local/Cellar/hbase/0.98.1/libexec"
export HIVE_HOME="/usr/local/Cellar/hive/0.13.0/libexec"
export ZOOCFGDIR="/usr/local/etc/zookeeper"
export HCAT_HOME="/usr/local/Cellar/hive/0.13.0/libexec/hcatalog"

and to answer your question the hbase directory has the following file /

CHANGES.txt LICENSE.txt README.txt libexec

Now, when I try and run my import I get a new error (and the hbase table is
actually created, just no data going into it)

The error is

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)

I've read this is because of incompatible jar files but, like I said I'm a
complete newb at all of this and if I could get a little more direction on
what exactly I'm missing / is incompatible that would be fantastic.

This is all on homebrew mac os x 10.9 hadoop 2.4 hbase .98 sqoop 1.4.4

On Thu, Jun 5, 2014 at 2:14 AM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Hey there,
> What does the contents of the directory '/usr/local/Cellar/hbase/0.98.1'
> look like? The command 'find /usr/local/Cellar/hbase/0.98.1' should
> assist in figuring that out. Please paste the result of that command here.
> The environment variables listed above are used in many different areas of
> Hadoop. I've typically seen them in startup scripts to resolve libraries
> and binaries. In this case, it's used to include jars in the classpath when
> you call Sqoop.
> Are you able to import to HDFS?
> -Abe
> On Wed, Jun 4, 2014 at 6:59 PM, Josh Millstein <joshua.millstein@gmail.com
> > wrote:
>> I've got hadoop / hbase / hive / set up and running (can create files on
>> hdfs, run map reduce jobs, create a "table" in hbase and also in hive) on
>> my mac with osx 10.9. I'm now trying to import data from a mysql table into
>> sqoop (using query, not table-name etc). I am getting this error with this
>> command.
>> sqoop import --connect jdbc:mysql:///joshLocal --username root --query "SELECT *
FROM     BITLOG WHERE \$CONDITIONS" --split-by oozie_job.id --hbase-table bitlogTest --hbase-create-
   table --column-family bitLogColumn
>> --ERROR
>> ERROR tool.ImportTool: Error during import: HBase jars are not present in classpath,
cannot import to HBase!
>>  I believe that all the export vars are correctly setup. I have the
>> following in swoop-env.sh
>> export HADOOP_HOME="/usr/local/Cellar/hadoop/2.4.0"
>> export HBASE_HOME="/usr/local/Cellar/hbase/0.98.1"
>> export HIVE_HOME="/usr/local/Cellar/hive/0.13.0"
>> export ZOOCFGDIR="/usr/local/etc/zookeeper"
>> export HCAT_HOME="/usr/local/Cellar/hive/0.13.0/libexec/hcatalog"
>>  one thing I did that gave a different message was to change the hbase
>> home to point to HBASE_HOME/libexec/lib in swoop-env.sh. That gave me a
>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/Tool
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>  error. I have seen some advice given that says I need to copy over
>> hadoop security jar files to hbase's installation. I don't exactly know
>> what files need to go over and if that is even an issue. The only reason I
>> thought it might be is because java.security.SecureClassLoader is in the
>> stacktrace.
>> I'm sorry if this is a really basic java question but I'm a complete
>> novice with it.
>> One other even more basic java question. When we define HADOOP_HOME
>> HBASE_HOME etc., what are we "telling" the other java programs that rely on
>> that info. Are we saying "here is the executable java file" or are we
>> saying "here are the jar file in the lib folder" . I don't quite understand
>> what I should actually be pointing to because I don't know how that path is
>> used.

View raw message