spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuanjian Li (JIRA)" <>
Subject [jira] [Commented] (SPARK-26225) Scan: track decoding time for row-based data sources
Date Tue, 25 Dec 2018 08:37:00 GMT


Yuanjian Li commented on SPARK-26225:

We define decoding time here as the time which the system cost on converting data from the
storage format to 'InternalRow' of Spark. I list decoding source code here and divide them
into two parts.
1. Row-based data sources
All decoding work happened in 'buildReader' function of row-based data sources, which override
from FileFormat.buildReader.
||Data Source||Decode Logic||Code Link||

Instead of dealing with all scenario separately, we can handle them uniformly by timing FileFormat.buildreader
if we can accept the initializing work(like reader initialization, schema preparation, etc)
count in decoding time. That can be more code and logical clean as well as overhead minimize.

2. Column-based data sources

All decoding work triggered in buildReaderWithPartitionValures which override from FileFormat, it
should discuss separately by batch read mode enable or disable.
||Data Source||Batch Read||Decode Logic||Code Link||
|ORC-native|true|Full fill column vector in OrcColumnBatchReader.nextBatch|[\|]|
|Parquet|false|InternalParquetRecordReader|This part of code not in Spark, the decoding work
is done in RecordMaterializer|
|Parquet|true|Full fill column vector in VectorizedColumnReader.readBatch|[\|]|

Listing decoding logic of column-based data sources, if further work is needed later.

> Scan: track decoding time for row-based data sources
> ----------------------------------------------------
>                 Key: SPARK-26225
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Reynold Xin
>            Priority: Major
> Scan node should report decoding time for each record, if it is not too much overhead.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message