drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5279) JsonRecordReader inefficient: allocates temp direct memory buffers
Date Mon, 20 Feb 2017 21:56:44 GMT
Paul Rogers created DRILL-5279:

             Summary: JsonRecordReader inefficient: allocates temp direct memory buffers
                 Key: DRILL-5279
                 URL: https://issues.apache.org/jira/browse/DRILL-5279
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.10.0
            Reporter: Paul Rogers
            Priority: Minor

The {{JsonRecordReader}} class uses the {{VarCharWriter}} class to add a string value to a
VarChar vector. The {{VarCharWriter}} method to add a string has this signature:

void writeVarChar(int start, int end, DrillBuf buffer);

This means that the {{JsonRecordReader}} has to convert the JSON string value to a direct
memory buffer, then write it into the buffer.

Direct memory buffers are expensive to create and should be done only to hold large amounts
of semi-persistent data (such as value vectors.)

Extend the {{VarCharWriter}} to accept a Java String and/or heap buffer to avoid having to
incur the overhead of the direct memory temporary buffer.

This message was sent by Atlassian JIRA

View raw message