sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-485) --hive-import cannot be used with --query
Date Sun, 20 May 2012 09:02:40 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279703#comment-13279703

Jarek Jarcec Cecho commented on SQOOP-485:

Hi Dave,
I'm using sqoop to do hive imports (both full and incremental) on daily basis, so I believe
that this functionality is working properly :-) I just tried to express that the usage is
little bit different.

I do have couple of notes to your extended variant of my workaround.

1. This command will create table's metadata in hive only once on first execution. Any metadata
changes to table on Oracle side (adding column, removing column, ...) won't be reflected (updated)
in hive on any of the subsequent calls.
2. Running sqoop with --append won't give you any extended functionality here as it appears
that you're importing into empty directory. Parameter --append make sense only in case that
you're doing sort of incremental import do directory that already contains some files.
3. -
4. As far as I know, hive do not allow you to execute this command if the table fact already
contains any data. You need to use "OVERRIDE" keyword, but then you'll loose previous content
of the table. I believe that this is not desired, right?

> --hive-import cannot be used with --query
> -----------------------------------------
>                 Key: SQOOP-485
>                 URL: https://issues.apache.org/jira/browse/SQOOP-485
>             Project: Sqoop
>          Issue Type: Bug
>          Components: hive-integration
>    Affects Versions: 1.3.0
>         Environment: :/home/w17626> uname -a
> Linux il93rhel91 2.6.18-128.2.1.el5 #1 SMP Wed Jul 8 11:54:47 EDT 2009 x86_64 x86_64
x86_64 GNU/Linux
>            Reporter: David Randolph
> sqoop import --verbose --connect jdbc:oracle:thin:@//oss-devdb2.pcs.mot.com/TOOLS --username
HADOOP --password xxx --query 'select * from fact where id < 11 and $CONDITIONS' --hive-import
--hive-table fact --fields-terminated-by , --escaped-by \\ -m1 --append
> produces the following output:
> {noformat}
> sqoop import --verbose --connect jdbc:oracle:thin:@//oss-devdb2.pcs.mot.com/TOOLS --username
HADOOP --password xxx --query 'select * from fact where id < 11 and $CONDITIONS' --hive-import
--hive-table fact --fields-terminated-by , --escaped-by \\ -m1 --append
> 12/05/15 11:10:10 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/05/15 11:10:10 WARN tool.BaseSqoopTool: Setting your password on the command-line
is insecure. Consider using -P instead.
> 12/05/15 11:10:10 DEBUG sqoop.Sqoop: Must specify destination with --target-dir.
> Try --help for usage instructions.
> Must specify destination with --target-dir.
> Try --help for usage instructions.
>         at com.cloudera.sqoop.tool.ImportTool.validateImportOptions(ImportTool.java:823)
>         at com.cloudera.sqoop.tool.ImportTool.validateOptions(ImportTool.java:890)
>         at com.cloudera.sqoop.Sqoop.run(Sqoop.java:132)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182)
>         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221)
>         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230)
>         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239)
> Must specify destination with --target-dir.
> Try --help for usage instructions.
> <snip>
> {noformat}
> I can work around this immediate failure by specifying the --target-dir flag. This seems
to do the right thing, but the sqoop command hangs. Also, use of --target-dir with --hive-import
is discouraged in SQOOP-464.
> How can I use the --query flag to import into Hive?
> :> sqoop version
> Sqoop 1.3.0-cdh3u3
> git commit id 57cbc8d38cc5ff22a24d34c3d13f9862fd1372bb
> Compiled by jenkins@ubuntu-slave02 on Thu Jan 26 10:41:26 PST 2012
> Thanks,
> Dave

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message