arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bowman <Brian.Bow...@sas.com>
Subject Re: Passing File Descriptors in the Low-Level API
Date Sun, 17 Mar 2019 00:47:05 GMT
Thanks Wes!

I'm working on the integrating and testing the necessary changes in our dev environment. 
I'll submit a PR once things are working.

Best,

Brian 

On 3/16/19, 4:24 PM, "Wes McKinney" <wesmckinn@gmail.com> wrote:

    EXTERNAL
    
    hi Brian,
    
    Please feel free to submit a PR to add the requisite APIs that you
    need for your application. Antoine or I or others should be able to
    give prompt feedback since we know this code pretty well.
    
    Thanks
    Wes
    
    On Sat, Mar 16, 2019 at 11:40 AM Brian Bowman <Brian.Bowman@sas.com> wrote:
    >
    > Hi Wes,
    >
    > Thanks for the quick reply!  To be clear, the usage I'm working on needs to own both
the Open FileDescriptor and corresponding mapped memory.  In other words ...
    >
    > SAS component does both open() and mmap() which could be for READ or WRITE.
    >
    > -> Calls low-level Parquet APIs to read an existing file or write a new one. 
The open() and mmap() flags are guaranteed to be correct.
    >
    > At some later point SAS component does an unmap() and close().
    >
    > -Brian
    >
    >
    > On 3/14/19, 3:42 PM, "Wes McKinney" <wesmckinn@gmail.com> wrote:
    >
    >     hi Brian,
    >
    >     This is mostly an Arrow platform question so I'm copying the Arrow mailing list.
    >
    >     You can open a file using an existing file descriptor using ReadableFile::Open
    >
    >     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.h#L145
    >
    >     The documentation for this function says:
    >
    >     "The file descriptor becomes owned by the ReadableFile, and will be
    >     closed on Close() or destruction."
    >
    >     If you want to do the equivalent thing, but using memory mapping, I
    >     think you'll need to add a corresponding API to MemoryMappedFile. This
    >     is more perilous because of the API requirements of mmap -- you need
    >     to pass the right flags and they may need to be the same flags that
    >     were passed when opening the file descriptor, see
    >
    >     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.cc#L378
    >
    >     and
    >
    >     https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/file.cc#L476
    >
    >     - Wes
    >
    >     On Thu, Mar 14, 2019 at 1:47 PM Brian Bowman <Brian.Bowman@sas.com> wrote:
    >     >
    >     >  The ReadableFile class (arrow/io/file.cc) has utility methods where a FileDescriptor
is either passed in or returned, but I don’t see how this surfaces through the API.
    >     >
    >     > Is there a way for application code to control the open lifetime of mmap()’d
Parquet files by passing an already open FileDescriptor to Parquet low-level API open/close
methods?
    >     >
    >     > Thanks,
    >     >
    >     > Brian
    >     >
    >
    >
    >
    

Mime
View raw message