sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua Stanfield (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-2282) Add validation check for --hive-import and --append
Date Thu, 30 Jul 2015 16:49:05 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647914#comment-14647914
] 

Joshua Stanfield edited comment on SQOOP-2282 at 7/30/15 4:48 PM:
------------------------------------------------------------------

Hey [~jarcec], our problem was that we had jobs stored in the metastore; we have a script
that iterates through the stored jobs and tries to run each one. Effectively our script does
{code}
 for job in $(sqoop job --list); do
   sqoop job --exec $job
 done
{code}

So after the upgrade, trying to run any stored job failed with the message [~aaditya.prodigy]
encountered.  
bq. Append mode for hive imports is not yet supported. Please remove the parameter --append-mode

I resolved by editing the metastore db script and setting all parameters where "hive.import"
is true to have "hdfs.append.dir" set to false. Effectively
{code}
update sqoop_sessions ss 
inner join (select job_name from sqoop_sessions where propname='hive.import' and propval='true')
jobs on jobs.job_name=ss.job_name
set ss.propval='false'
where 
 ss.propname='hdfs.append.dir'
{code}

Note - I didn't actually run this exact script; I hand edited the shared.db.script file itself.
But thought the script above might come in handy for someone else. 

After editing the metastore and bringing it back up, everything worked fine. 

fwiw - I looked back at our old commits, and {{ --append }} has been used for every stored
job in our repo dating back to 2010; I think there was some confusion at the time as to what
{{ --append }} did at the time, as (iirc) it predates the {{--incremental=append }} option.



was (Author: p5k6@yahoo.com):
Hey [~jarcec], our problem was that we had jobs stored in the metastore; we have a script
that iterates through the stored jobs and tries to run each one. Effectively our script does
{code}
 for job in $(sqoop job --list); do
   sqoop job --exec $job
 done
{code}

So after the upgrade, trying to run any stored job failed with the message [~aaditya.prodigy]
encountered.  
bq. Append mode for hive imports is not yet supported. Please remove the parameter --append-mode

I resolved by editing the metastore db script and setting all parameters where "hive.import"
is true to have "hdfs.append.dir" set to false. Effectively
{code}
update sqoop_sessions ss 
inner join (select job_name from sqoop_sessions where propname='hive.import' and propval='true')
jobs on jobs.job_name=ss.job_name
set ss.propval='false'
where 
 ss.propname='hdfs.append.dir'
{code}

Note - I didn't actually run this exact script; I hand edited the shared.db.script file itself.
But thought the script above might come in handy for someone else. 

After editing the metastore and bringing it back up, everything worked fine. 

fwiw - I looked back at our old commits, and {noformat} --append {noformat} has been used
for every stored job in our repo dating back to 2010; I think there was some confusion at
the time as to what {noformat} --append {noformat} did at the time, as (iirc) it predates
the {noformat} --incremental=append {noformat} option. 

> Add validation check for --hive-import and --append
> ---------------------------------------------------
>
>                 Key: SQOOP-2282
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2282
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Abraham Elmahrek
>            Assignee: Abraham Elmahrek
>             Fix For: 1.4.6
>
>         Attachments: SQOOP-2282.patch
>
>
> They don't work together currently. Though, it seems easy to add.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message