ISHAN gaur created SQOOP-3477:
---------------------------------
Summary: Sqoop-import creating .metadata and .signal folders while import
Key: SQOOP-3477
URL: https://issues.apache.org/jira/browse/SQOOP-3477
Project: Sqoop
Issue Type: Bug
Components: sqoop2-client
Affects Versions: 1.4.7
Reporter: ISHAN gaur
Fix For: 1.4.7
Unable to read sqoop-import extracted files in hive
Steps to reproduce -
-1- Import a table from a relational database( Oracle RDS ) or any other rdbms into S3 through
sqoop import.
Sqoop-import Sample -
sqoop import --connect jdbc:oracle:thin:@//ishantest.xxx.us-west-2.rds.amazonaws.com[|https://rds-op-usw2.amazon.com/ops/search?query=ishantest#linker_verify_account=515450464085]:1521/ISHANORC
--table PV_AGGREGATES --username admin --password xxx# -m 1 --delete-target-dir --target-dir
s3n://xx-isgxaur-logs/sqoop_issue/pv_ag --as-parquetfile --compression-codec=snappy
Once sqoop-import is done - Data in s3 will look like -
[hadoop@ip-xx-xx-xx-xx ~]$ aws s3 ls s3://xx-xx-xx/sqoop_issue/ishan_new/
PRE .metadata/
PRE .signals/
2020-06-03 17:49:03 0 .metadata_$folder$
2020-06-03 17:49:03 0 .signals_$folder$
2020-06-03 17:49:03 417 f7e5670c-77ca-460b-80c5-e0449cb09dfe.parquet
As we can see sqoop-import is cretaing .metadata and .signal folders while extracting the
data . This causes failure to read the data in hive
-2- Deploy a v5.19 EMR or any virtual machine with Hive 2.3.3 and sqoop 1.4.7 installed.
-3- Create an external table, whose location should be the path copied in point 1.
Command to create external table -
CREATE EXTERNAL TABLE test (dt string,code string,views string) STORED as parquet location
's3://xxx-isgaur-logs/sqoop_issue/ishan_new';
-4- Use hive and run a "select * from table; " statement to the table. you will get below
error
Error message it throws - Failed with exception java.io.IOException:java.lang.RuntimeException:
s3://cccxxxx-logs/sqoop_issue/ishan_new/.metadata/descriptor.properties is not a Parquet file.
expected magic number at tail [80, 65, 82, 49] but found [117, 101, 116, 10]
The above error message is because while reading the data in hive , it doesn't expect the
.metadata and .signal folders created by sqoop ,hence the failure.
I did extensive analysis on this problem but did not find any solution , please let me know
if there is any workaround to it ? How can one read the sqoop-import extracted data within
hive ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
|