sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ISHAN gaur (Jira)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-3477) Sqoop-import creating .metadata and .signal folders while import
Date Wed, 03 Jun 2020 20:29:00 GMT
ISHAN gaur created SQOOP-3477:

             Summary: Sqoop-import creating .metadata and .signal folders while import
                 Key: SQOOP-3477
                 URL: https://issues.apache.org/jira/browse/SQOOP-3477
             Project: Sqoop
          Issue Type: Bug
          Components: sqoop2-client
    Affects Versions: 1.4.7
            Reporter: ISHAN gaur
             Fix For: 1.4.7

Unable to read sqoop-import extracted files in hive 


Steps to reproduce -

-1- Import a table from a relational database( Oracle RDS ) or any other rdbms into S3 through
sqoop import.

Sqoop-import Sample -

sqoop import --connect jdbc:oracle:thin:@//ishantest.xxx.us-west-2.rds.amazonaws.com[|https://rds-op-usw2.amazon.com/ops/search?query=ishantest#linker_verify_account=515450464085]:1521/ISHANORC
--table PV_AGGREGATES --username admin --password xxx# -m 1 --delete-target-dir --target-dir
s3n://xx-isgxaur-logs/sqoop_issue/pv_ag --as-parquetfile --compression-codec=snappy

Once sqoop-import is done - Data in s3 will look like -

[hadoop@ip-xx-xx-xx-xx ~]$ aws s3 ls s3://xx-xx-xx/sqoop_issue/ishan_new/

PRE .metadata/
PRE .signals/
2020-06-03 17:49:03 0 .metadata_$folder$
2020-06-03 17:49:03 0 .signals_$folder$
2020-06-03 17:49:03 417 f7e5670c-77ca-460b-80c5-e0449cb09dfe.parquet


As we can see sqoop-import is cretaing .metadata and .signal folders while extracting the
data . This causes failure to read the data in hive

-2- Deploy a v5.19 EMR or any virtual machine with Hive 2.3.3 and sqoop 1.4.7 installed.

-3- Create an external table, whose location should be the path copied in point 1.

Command to create external table -

CREATE EXTERNAL TABLE test (dt string,code string,views string) STORED as parquet location

-4- Use hive and run a "select * from table; " statement to the table. you will get below

Error message it throws - Failed with exception java.io.IOException:java.lang.RuntimeException:
s3://cccxxxx-logs/sqoop_issue/ishan_new/.metadata/descriptor.properties is not a Parquet file.
expected magic number at tail [80, 65, 82, 49] but found [117, 101, 116, 10]

The above error message is because while reading the data in hive , it doesn't expect the
.metadata and .signal folders created by sqoop ,hence the failure.

I did extensive analysis on this problem but did not find any solution , please let me know
if there is any workaround to it ? How can one read the sqoop-import extracted data within
hive ?

This message was sent by Atlassian Jira

View raw message