spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Blue (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-28878) DataSourceV2 should not insert extra projection for columnar batches
Date Mon, 26 Aug 2019 22:41:00 GMT
Ryan Blue created SPARK-28878:
---------------------------------

             Summary: DataSourceV2 should not insert extra projection for columnar batches
                 Key: SPARK-28878
                 URL: https://issues.apache.org/jira/browse/SPARK-28878
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Ryan Blue


SPARK-23325 added an extra physical projection when reading from a DSv2 source because some
Spark operators assume that InternalRow instances are actually UnsafeRow. The projection ensures
that InternalRow is converted to UnsafeRow. This isn't needed for the columnar batch read
path because this is already done when converting from columnar operators to row-based operators
in InputRDDCodegen.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message