sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aravindakshan Srinivasan <s_arav...@yahoo.com>
Subject Re: Oozie + Sqoop + MySql UPSERT
Date Fri, 08 May 2015 18:39:41 GMT
Wow, it did work. Why though?
Thanks a lot Abraham.
Regards,Aravind --------------------------------------------------------------------------------------------Aravind
Srinivasan 

http://goodcynicsports.blogspot.com

I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.--------------------------------------------------------------------------------------------

      From: Abraham Elmahrek <abe@cloudera.com>
 To: "user@sqoop.apache.org" <user@sqoop.apache.org>; Aravindakshan Srinivasan <s_aravind@yahoo.com>

 Sent: Friday, May 8, 2015 11:52 PM
 Subject: Re: Oozie + Sqoop + MySql UPSERT
   
I'd get rid of --driver and see if that works.


On Fri, May 8, 2015 at 11:14 AM, Aravindakshan Srinivasan <s_aravind@yahoo.com> wrote:

Hi Abraham,
Here's the Sqoop command that works.
sqoop export --connect jdbc:mysql://myhost.rds.amazonaws.com/DB1 --username XXX -password
YYY --table test_table --columns "name,ts,age,salary" --update-key "name" --export-dir /user/aravind/test/year=2015/month=5/day=7/hour=19
--fields-terminated-by "\t" --update-mode allowinsert -m 1
The following workflow does not work.
<workflow-app xmlns="uri:oozie:workflow:0.2" name="ara_sqoop-wf">  <start to="ara_move_to_mysql"/> 
<action name="ara_move_to_mysql">    <sqoop xmlns="uri:oozie:sqoop-action:0.3">
<job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node>
<arg>export</arg> <arg>-libjars</arg> <arg>${wf_folder}lib/mysql-connector-java.jar</arg>
<arg>--connect</arg> <arg>jdbc:mysql://myhost.rds.amazonaws.com/${mysql_db}</arg>
<arg>--driver</arg> <arg>com.mysql.jdbc.Driver</arg> <arg>--username</arg>
<arg>${mysql_usern}</arg> <arg>-password</arg> <arg>${mysql_pswd}</arg>
<arg>--table</arg> <arg>${table_name}</arg> <arg>--columns</arg>
<arg>"name,ts,age,salary"</arg> <arg>--update-key</arg> <arg>"name"</arg>
<arg>--export-dir</arg> <arg>/user/aravind/test/year=2015/month=5/day=7/hour=19</arg>
<arg>--fields-terminated-by</arg> <arg>"\t"</arg> <arg>--update-mode</arg>
<arg>allowinsert</arg> <arg>-m</arg> <arg>1</arg>  
</sqoop>   <ok to="end"/>   <error to="fail"/>  </action>  <kill
name="fail">      <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
</kill>  <end name="end"/></workflow-app>
Thanks,Aravind
      From: Abraham Elmahrek <abe@cloudera.com>
 To: "user@sqoop.apache.org" <user@sqoop.apache.org>; Aravindakshan Srinivasan <s_aravind@yahoo.com>

 Sent: Friday, May 8, 2015 10:36 PM
 Subject: Re: Oozie + Sqoop + MySql UPSERT
   
Can you send over your oozie job or sqoop command?


On Fri, May 8, 2015 at 8:30 AM, Aravindakshan Srinivasan <s_aravind@yahoo.com> wrote:

Friends,

I have an odd problem. I am trying to run an MySql upsert via Sqoop. It works fine from the
edge node that has Sqoop 1.4.5 installed. When I try to run via Oozie, I get this error.
org.apache.sqoop.tool.ExportTool - Error during export: Mixed update/insert is not supported
against the target database
This feature is available in the newer versions of Sqoop starting with Sqoop 1.4.3, which
is what my Oozie sharelib contains. Still, my first reaction was that the Sqoop jars in Oozie's
sharelib are different and hence throwing this error. As you know, the Sqoop via Oozie runs
on the Hadoop compute nodes. I tried copying the Sqoop jars from my edge node to the workflow/lib
dir. It still didn't work. 


Since then, I have tried everything I can think of, but can't get past this error. I have
even rebuilt the Oozie sharelib with Sqoop 1.4.5 and tried both  oozie.libpath and oozie.action.sharelib.for.sqoop
pointing to my rebuilt lib. Still no luck. What am I missing?
Thanks,Aravind --------------------------------------------------------------------------------------------Aravind
Srinivasan 

http://goodcynicsports.blogspot.com

I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.--------------------------------------------------------------------------------------------
   

   



   



  
Mime
View raw message