drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maarten D'Haene (Jira)" <j...@apache.org>
Subject [jira] [Created] (DRILL-7762) Parquet files with too many columns generated in Python (pyarrow, pandas) are not readable
Date Wed, 01 Jul 2020 14:14:00 GMT
Maarten D'Haene created DRILL-7762:

             Summary: Parquet files with too many columns generated in Python (pyarrow, pandas)
are not readable  
                 Key: DRILL-7762
                 URL: https://issues.apache.org/jira/browse/DRILL-7762
             Project: Apache Drill
          Issue Type: Bug
          Components: Functions - Drill, SQL Parser, Storage - Parquet
    Affects Versions: 1.17.0
            Reporter: Maarten D'Haene
             Fix For: Future
         Attachments: error_drill_parquet.doc, shape_file_snappy512.parquet

When launching a query 
|SELECT * FROM s3.datascience.`./government/shape_file_snappy512.parquet` |

on a parquet-file with too many columns generated in Python, I get following error: 

User Error Occurred: Error in drill parquet reader (complex). Message: Failure in setting
up reader Parquet Metadata: ParquetMetaData\{FileMetaData{schema: message schema { optional
int64 OBJECTID_1; optional int64 OBJECTID; optional binary Cs012011 (UTF8); optional double
Nis_012011; optional binary Sec012011 (UTF8); optional binary CS102001 (UTF8); optional binary
CS031991 (UTF8); optional binary CS031981 (UTF8); optional binary Sector_nl (UTF8); optional
binary Sector_fr (UTF8); optional binary Gemeente (UTF8); optional binary Commune (UTF8);
optional binary Arrond_nl (UTF8); optional binary Arrond_fr (UTF8); optional binary Prov_nl
(UTF8); optional binary Prov_fr (UTF8); optional binary Reg_nl (UTF8); optional binary Reg_fr
(UTF8); optional binary Nuts1 (UTF8); optional binary Nuts2 (UTF8); optional binary Nuts3_new
(UTF8); optional int64 Inhab; optional double Gis_Perime; optional double Gis_area_h; optional
double Cad_area_h; optional double Shape_Leng; optional double Shape_Area; optional binary
codesecteu (UTF8); optional binary CD_REFNIS (UTF8); optional binary CD_SECTOR (UTF8); optional
double TOTAL; optional double MALES; optional double FEMALES; optional double group0_14; optional
double group15_64; optional double group65ETP; optional binary areaofdis (UTF8); }

The parquet file is generated using pyarrow with compression codec 'snappy' and data page
size 512MB. Smaller/bigger page sizes give same error. The files originate on on-premise s3
object store (dell ecs). Other queries on the same parquet-file (count(*), select OBJECTID_1
from .. ) succeed succesfully. Doing a 'select *' on a parquet-file with less columns generated
the same way also run without any issues. A workaround is to export a csv-file from Python
and generate the parquet file using Drill itself starting from this csv-file (CREATE TABLE
s3.datascience.`./government/tes3` AS SELECT * FROM s3.datascience.`./government/shape_file.csv`).
Querying a parquet-file generated this way don't result in any problems (although content
is exactly the same as parquet-file generated in Python). Is there an explanation why Drill
acts this way and what are the specifications of the parquet-file generated by Drill itself
(so we can aim to match these specification when creating a parquet-file using Pyarrow/Pandas)?

This message was sent by Atlassian Jira

View raw message