drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Padma Penumarthy <ppenumar...@mapr.com>
Subject Re: Which perform better JSON or convert JSON to parquet format ?
Date Mon, 11 Jun 2018 04:36:21 GMT
Yes, parquet is always better for multiple reasons. With JSON, we have to read the whole file
from a single reader thread and have to parse to read individual columns. 
Parquet compresses and encodes data on disk. So, we read much less data from disk.
Drill can read individual columns with in each rowgroup in parallel. Also, we can leverage
features like filter pushdown, partition pruning, metadata cache for better query performance.


Thanks
Padma

> On Jun 10, 2018, at 8:22 PM, Abhishek Girish <agirish@apache.org> wrote:
> 
> I would suggest converting the JSON files to parquet for better
> performance. JSON supports a more free form data model, so that's a
> trade-off you need to consider, in my opinion.
> On Sun, Jun 10, 2018 at 8:08 PM Divya Gehlot <divya.htconex@gmail.com>
> wrote:
> 
>> Hi,
>> I am looking for the advise regarding the performance for below :
>> 1. keep the JSON as is
>> 2. Convert the JSON file to parquet files
>> 
>> My JSON files data is not in fixed format and  file size varies from 10 KB
>> to 1 MB.
>> 
>> Appreciate the community users advise on above !
>> 
>> 
>> Thanks,
>> Divya
>> 


Mime
View raw message