parquet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: Passing File Descriptors in the Low-Level API
Date Sat, 16 Mar 2019 20:24:12 GMT
hi Brian,

Please feel free to submit a PR to add the requisite APIs that you
need for your application. Antoine or I or others should be able to
give prompt feedback since we know this code pretty well.

Thanks
Wes

On Sat, Mar 16, 2019 at 11:40 AM Brian Bowman <Brian.Bowman@sas.com> wrote:
>
> Hi Wes,
>
> Thanks for the quick reply!  To be clear, the usage I'm working on needs to own both
the Open FileDescriptor and corresponding mapped memory.  In other words ...
>
> SAS component does both open() and mmap() which could be for READ or WRITE.
>
> -> Calls low-level Parquet APIs to read an existing file or write a new one.  The
open() and mmap() flags are guaranteed to be correct.
>
> At some later point SAS component does an unmap() and close().
>
> -Brian
>
>
> On 3/14/19, 3:42 PM, "Wes McKinney" <wesmckinn@gmail.com> wrote:
>
>     hi Brian,
>
>     This is mostly an Arrow platform question so I'm copying the Arrow mailing list.
>
>     You can open a file using an existing file descriptor using ReadableFile::Open
>
>     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.h#L145
>
>     The documentation for this function says:
>
>     "The file descriptor becomes owned by the ReadableFile, and will be
>     closed on Close() or destruction."
>
>     If you want to do the equivalent thing, but using memory mapping, I
>     think you'll need to add a corresponding API to MemoryMappedFile. This
>     is more perilous because of the API requirements of mmap -- you need
>     to pass the right flags and they may need to be the same flags that
>     were passed when opening the file descriptor, see
>
>     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.cc#L378
>
>     and
>
>     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.cc#L476
>
>     - Wes
>
>     On Thu, Mar 14, 2019 at 1:47 PM Brian Bowman <Brian.Bowman@sas.com> wrote:
>     >
>     >  The ReadableFile class (arrow/io/file.cc) has utility methods where a FileDescriptor
is either passed in or returned, but I don’t see how this surfaces through the API.
>     >
>     > Is there a way for application code to control the open lifetime of mmap()’d
Parquet files by passing an already open FileDescriptor to Parquet low-level API open/close
methods?
>     >
>     > Thanks,
>     >
>     > Brian
>     >
>
>
>

Mime
View raw message