parquet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tham (JIRA)" <>
Subject [jira] [Commented] (PARQUET-1022) [C++] Append mode in parquet-cpp
Date Wed, 13 Mar 2019 02:16:00 GMT


Tham commented on PARQUET-1022:

Thanks for your suggestion [~xhochy] [~wesmckinn]. I'm thinking between
 * writing multiple files and merge it into a new single file later (but it seems to take
long time to merge)
 * or write row groups separately from metadata and concat into a single file later.

Is there any feature of merging/concating files on C++?

> [C++] Append mode in parquet-cpp
> --------------------------------
>                 Key: PARQUET-1022
>                 URL:
>             Project: Parquet
>          Issue Type: New Feature
>          Components: parquet-cpp
>    Affects Versions: cpp-1.1.0
>            Reporter: yugu
>            Assignee: Wes McKinney
>            Priority: Major
> As said, currently trying to work out a append feature for parquet files in c++.
> (been searching through repo etc, can't find example tho..)
> Current solution is to (assume no schema changes that is):
> Read in metadata
> Change metadata based on appended rows+ original rows
> Append a new row group (or multiple row group writer)
> Write the new rows.
> ---
> The problem is that, is approached this way, the original last row group may not be complete
filled. Was wondering if there is a fix or I'm using the api wrong...
> Thanks ! : D

This message was sent by Atlassian JIRA

View raw message