sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vasanth kumar RJ (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SQOOP-951) --export-dir to support subdirectories
Date Mon, 15 Jul 2013 17:40:52 GMT

     [ https://issues.apache.org/jira/browse/SQOOP-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vasanth kumar RJ updated SQOOP-951:
-----------------------------------

    Fix Version/s:     (was: 1.4.4)
                   1.4.5
    
> --export-dir to support subdirectories
> --------------------------------------
>
>                 Key: SQOOP-951
>                 URL: https://issues.apache.org/jira/browse/SQOOP-951
>             Project: Sqoop
>          Issue Type: Improvement
>    Affects Versions: 1.4.3
>         Environment: Debian GNU/Linux 6.0
>            Reporter: Matthieu Labour
>            Assignee: Vasanth kumar RJ
>             Fix For: 1.4.5
>
>         Attachments: SQOOP-951.patch
>
>
> I am using sqoop-1.4.2 to export to Sql.
> --export-dir does not work when the dir being passed is the root of subdirectories. -export-dir
is not doing any recursive lookup for files. It expects directory with files that you want
export.
> It would be great if one could pass a directory with subdirectories.
> Example:
> The following command exports the data to Sql
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx
--username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir
=hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01 --input-fields-terminated-by='\t'
--lines-terminated-by='\n' --verbose --batch
> hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
> Found 1 items
> -rw-r--r--   1 hadoop supergroup   15931406 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01/part-r-00001
> The following command does not export the data to Sql
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx
--username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir
=hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized --input-fields-terminated-by='\t' --lines-terminated-by='\n'
--verbose --batch
> hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/
> Found 44 items
> -rw-r--r--   1 hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/_SUCCESS
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-02
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-03
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-04
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-05
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-06

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message