carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipes...@apache.org
Subject [carbondata] branch branch-1.5 updated (4f95559 -> ea1e86c)
Date Thu, 16 May 2019 19:05:33 GMT
This is an automated email from the ASF dual-hosted git repository.

ravipesala pushed a change to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/carbondata.git.


    from 4f95559  [maven-release-plugin] prepare for next development iteration
     new 9a54c88  [CARBONDATA-3341] fixed invalid NULL result in filter query
     new eeb7e3a  [CARBONDATA-3001] configurable page size in MB
     new f0e2706  [CARBONDATA-3331] Fix for external table in Show Metacache
     new d1bb3a0  [CARBONDATA-3334] fixed multiple segment file issue for partition
     new 743b843  [CARBONDATA-3353 ]Fixed MinMax Based Pruning for Measure column in case
of Legacy store
     new 4f7f17d  [CARBONDATA-3344] Fix Drop column not present in table
     new 69b8873  [CARBONDATA-3351] Support Binary Data Type
     new d1b455f  [CARBONDATA-3348] Support alter SORT_COLUMNS property
     new 9a9c791  [CARBONDATA-3353 ][HOTFIX]Fixed MinMax Based Pruning for Measure column
in case of Legacy store
     new 9f23d2c  [HOTFIX] support compact segments with different sort_columns
     new f7cdb47  [CARBONDATA-3359]Fix data mismatch issue for decimal column after delete
operation
     new f80a28d  [CARBONDATA-3345]A growing streaming ROW_V1 carbondata file would be ingored
some InputSplits
     new 7449c34  [CARBONDATA-3343] Compaction for Range Sort
     new bc80a22  [CARBONDATA-3360]fix NullPointerException in delete and clean files operation
     new f46ad43  [CARBONDATA-3369] Fix issues during concurrent execution of Create table
If not exists
     new d8b0ff4  [CARBONDATA-3371] Fix ArrayIndexOutOfBoundsException of compaction after
sort_columns modification
     new 7e7792e  [CARBONDATA-3375] [CARBONDATA-3376] Fix GC Overhead limit exceeded issue
and partition column as range column issue
     new 4d21b6a  [DOC] Update doc for sort_columns modification
     new 251cbdc  [CARBONDATA-3362] Document update for pagesize table property scenario
     new b42f1ac  [CARBONDATA-3374] Optimize documentation and fix some spell errors.
     new 4abed04  [CARBONDATA-3377] Fix for Null pointer exception in Range Col compaction
     new ea1e86c  [CARBONDATA-3391] Count star output is wrong when BLOCKLET CACHE is enabled

The 22 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/constants/CarbonCommonConstants.java      |   25 +
 .../carbondata/core/datamap/DataMapFilter.java     |   89 ++
 .../carbondata/core/datamap/TableDataMap.java      |   91 +-
 .../core/datamap/dev/DataMapFactory.java           |    4 +-
 .../datamap/dev/expr/DataMapExprWrapperImpl.java   |    3 +-
 .../block/SegmentPropertiesAndSchemaHolder.java    |   13 +-
 .../core/datastore/blocklet/EncodedBlocklet.java   |   19 +
 .../datastore/chunk/store/ColumnPageWrapper.java   |    7 +-
 .../safe/AbstractNonDictionaryVectorFiller.java    |    2 +-
 .../SafeVariableLengthDimensionDataChunkStore.java |    2 +-
 .../carbondata/core/datastore/page/ColumnPage.java |   11 +-
 .../core/datastore/page/LazyColumnPage.java        |    2 +
 .../datastore/page/UnsafeVarLengthColumnPage.java  |    7 +-
 .../datastore/page/VarLengthColumnPageBase.java    |    1 +
 .../page/encoding/ColumnPageEncoderMeta.java       |    4 +-
 .../page/encoding/DefaultEncodingFactory.java      |   14 +-
 .../carbondata/core/datastore/row/CarbonRow.java   |    6 +-
 .../core/indexstore/BlockletDetailsFetcher.java    |    4 +-
 .../indexstore/blockletindex/BlockDataMap.java     |   23 +-
 .../blockletindex/BlockletDataMapFactory.java      |    2 +-
 .../blockletindex/BlockletDataRefNode.java         |    6 +-
 .../core/indexstore/schema/SchemaGenerator.java    |    2 +-
 .../core/metadata/blocklet/BlockletInfo.java       |   10 +
 .../ThriftWrapperSchemaConverterImpl.java          |    4 +
 .../datatype/{StringType.java => BinaryType.java}  |   14 +-
 .../core/metadata/datatype/DataType.java           |    2 +-
 .../core/metadata/datatype/DataTypes.java          |    4 +
 .../metadata/datatype/DecimalConverterFactory.java |   55 +-
 .../core/metadata/schema/table/CarbonTable.java    |   44 +-
 .../core/metadata/schema/table/TableInfo.java      |   23 +
 .../metadata/schema/table/TableSchemaBuilder.java  |   11 +
 .../carbondata/core/mutate/CarbonUpdateUtil.java   |   48 +-
 .../scan/executor/impl/AbstractQueryExecutor.java  |   83 +-
 .../executor/impl/QueryExecutorProperties.java     |    5 -
 .../core/scan/executor/util/QueryUtil.java         |    2 +-
 .../core/scan/executor/util/RestructureUtil.java   |   77 +-
 .../core/scan/expression/Expression.java           |   13 +
 .../scan/filter/FilterExpressionProcessor.java     |    5 +-
 .../carbondata/core/scan/filter/FilterUtil.java    |   60 +-
 .../filter/executer/IncludeFilterExecuterImpl.java |    9 +-
 .../RowLevelRangeGrtThanFiterExecuterImpl.java     |   10 +-
 ...LevelRangeGrtrThanEquaToFilterExecuterImpl.java |   10 +-
 ...wLevelRangeLessThanEqualFilterExecuterImpl.java |   10 +-
 .../RowLevelRangeLessThanFilterExecuterImpl.java   |   10 +-
 .../resolver/ConditionalFilterResolverImpl.java    |    2 +-
 .../resolver/RowLevelRangeFilterResolverImpl.java  |   40 +-
 .../core/scan/model/QueryModelBuilder.java         |   20 +-
 .../core/scan/result/BlockletScannedResult.java    |   62 +-
 .../scan/result/impl/FilterQueryScannedResult.java |   20 +-
 .../result/impl/NonFilterQueryScannedResult.java   |   59 +-
 .../iterator/ColumnDriftRawResultIterator.java     |  128 +++
 .../scan/result/iterator/RawResultIterator.java    |   12 +-
 .../result/vector/impl/CarbonColumnVectorImpl.java |    6 +-
 .../core/scan/wrappers/ByteArrayWrapper.java       |    3 +
 .../carbondata/core/util/CarbonMetadataUtil.java   |   15 +-
 .../carbondata/core/util/CarbonProperties.java     |    6 +
 .../apache/carbondata/core/util/CarbonUtil.java    |   56 +-
 .../core/util/DataFileFooterConverterV3.java       |    8 +
 .../apache/carbondata/core/util/DataTypeUtil.java  |   11 +-
 .../carbondata/core/util/path/CarbonTablePath.java |    2 +-
 .../scan/executor/util/RestructureUtilTest.java    |   11 +-
 .../carbondata/core/util/CarbonUtilTest.java       |   46 -
 .../datamap/lucene/LuceneDataMapFactoryBase.java   |    2 +-
 .../lucene/LuceneFineGrainDataMapFactory.java      |    2 +-
 docs/carbon-as-spark-datasource-guide.md           |    3 +-
 docs/ddl-of-carbondata.md                          |   51 +-
 docs/sdk-guide.md                                  |    2 +
 docs/supported-data-types-in-carbondata.md         |    2 +-
 format/src/main/thrift/carbondata.thrift           |    1 +
 format/src/main/thrift/schema.thrift               |    1 +
 .../carbondata/hadoop/api/CarbonInputFormat.java   |   29 +-
 .../hadoop/util/CarbonVectorizedRecordReader.java  |    3 +-
 .../carbondata/presto/impl/CarbonTableReader.java  |    9 +-
 .../cluster/sdv/generated/AlterTableTestCase.scala |   12 +
 integration/spark-common-test/pom.xml              |    1 -
 .../org/apache/carbondata/sdk/util/BinaryUtil.java |   88 ++
 .../src/test/resources/binaryStringNullData.csv    |    4 +
 .../src/test/resources/binarydata.csv              |    3 +
 .../src/test/resources/binarystringdata.csv        |    3 +
 .../src/test/resources/binarystringdata2.csv       |    3 +
 .../src/test/resources/decimalData.csv             |    4 +
 .../resources/jsonFiles/data/allPrimitiveType.json |    3 +-
 .../test/resources/sort_columns/alldatatype1.csv   |   13 +
 .../test/resources/sort_columns/alldatatype2.csv   |   13 +
 .../testsuite/binary/TestBinaryDataType.scala      | 1153 ++++++++++++++++++++
 .../testsuite/dataload/TestLoadDataGeneral.scala   |    4 +-
 ...ryWithColumnMetCacheAndCacheLevelProperty.scala |    2 -
 .../TestAlterTableSortColumnsProperty.scala        |  573 ++++++++++
 .../createTable/TestCreateTableIfNotExists.scala   |   36 +
 .../TestCreateTableWithPageSizeInMb.scala          |   67 ++
 .../TestNonTransactionalCarbonTable.scala          |   49 +
 .../TestNonTransactionalCarbonTableForBinary.scala |  162 +++
 ...TestNonTransactionalCarbonTableJsonWriter.scala |   37 +-
 .../dataload/TestRangeColumnDataLoad.scala         |  734 ++++++++++++-
 .../testsuite/datamap/CGDataMapTestCase.scala      |    4 +-
 .../testsuite/datamap/DataMapWriterSuite.scala     |    2 +-
 .../testsuite/datamap/FGDataMapTestCase.scala      |    4 +-
 .../testsuite/iud/DeleteCarbonTableTestCase.scala  |   17 +
 .../testsuite/sortcolumns/TestSortColumns.scala    |   14 +
 .../StandardPartitionBadRecordLoggerTest.scala     |    2 -
 .../StandardPartitionTableLoadingTestCase.scala    |    8 +
 .../org/apache/carbondata/spark/CarbonOption.scala |    2 +
 .../spark/load/DataLoadProcessBuilderOnSpark.scala |   51 +-
 .../carbondata/spark/rdd/CarbonMergerRDD.scala     |  277 ++++-
 .../carbondata/spark/rdd/CarbonScanRDD.scala       |   17 +-
 .../spark/rdd/NewCarbonDataLoadRDD.scala           |    2 +-
 .../carbondata/spark/rdd/StreamHandoffRDD.scala    |    2 +-
 .../carbondata/spark/util/CarbonScalaUtil.scala    |   21 +-
 .../apache/carbondata/spark/util/CommonUtil.scala  |  112 +-
 .../spark/util/DataTypeConverterUtil.scala         |    3 +
 .../org/apache/spark/CarbonInputMetrics.scala      |    0
 .../apache/spark/DataSkewRangePartitioner.scala    |   26 +-
 .../spark/sql/catalyst/CarbonDDLSqlParser.scala    |   87 +-
 integration/spark-datasource/pom.xml               |    1 -
 .../converter/SparkDataTypeConverterImpl.java      |    5 +-
 .../vectorreader/VectorizedCarbonRecordReader.java |    2 +-
 .../datasources/CarbonSparkDataSourceUtil.scala    |   14 +-
 .../datasources/SparkCarbonFileFormat.scala        |    5 +-
 .../apache/spark/sql/util/SparkTypeConverter.scala |    1 +
 .../org/apache/carbondata/sdk/util/BinaryUtil.java |   89 ++
 .../SparkCarbonDataSourceBinaryTest.scala          |  544 +++++++++
 .../datasource/SparkCarbonDataSourceTest.scala     |   26 +
 .../apache/spark/sql/CarbonCatalystOperators.scala |    4 +-
 .../apache/spark/sql/CarbonDataFrameWriter.scala   |    1 +
 .../spark/sql/CarbonDatasourceHadoopRelation.scala |    1 -
 .../command/cache/CarbonShowCacheCommand.scala     |   91 +-
 .../command/management/CarbonLoadDataCommand.scala |    5 +-
 .../schema/CarbonAlterTableDropColumnCommand.scala |   14 +-
 .../command/table/CarbonCreateTableCommand.scala   |   33 +-
 .../table/CarbonDescribeFormattedCommand.scala     |    9 +-
 .../apache/spark/sql/optimizer/CarbonFilters.scala |   10 +-
 .../org/apache/spark/util/AlterTableUtil.scala     |  126 ++-
 .../sql/CarbonGetTableDetailComandTestCase.scala   |    0
 .../processing/datatypes/ArrayDataType.java        |   15 +
 .../processing/datatypes/GenericDataType.java      |    5 +
 .../processing/datatypes/PrimitiveDataType.java    |    6 +
 .../processing/datatypes/StructDataType.java       |   14 +
 .../converter/impl/BinaryFieldConverterImpl.java   |   78 ++
 .../converter/impl/FieldEncoderFactory.java        |    2 +
 .../loading/sort/SortStepRowHandler.java           |    9 +
 .../merger/CarbonCompactionExecutor.java           |   43 +-
 .../processing/merger/CarbonCompactionUtil.java    |  179 ++-
 .../merger/RowResultMergerProcessor.java           |    6 +-
 .../store/CarbonFactDataHandlerColumnar.java       |  191 +++-
 .../store/CarbonFactDataHandlerModel.java          |  106 +-
 .../carbondata/processing/store/TablePage.java     |   46 +-
 .../writer/v3/CarbonFactDataWriterImplV3.java      |    7 +
 store/sdk/pom.xml                                  |    3 +-
 .../carbondata/sdk/file/CSVCarbonWriter.java       |    2 +-
 .../apache/carbondata/sdk/file/CarbonReader.java   |    1 -
 .../carbondata/sdk/file/CarbonWriterBuilder.java   |   92 +-
 .../java/org/apache/carbondata/sdk/file/Field.java |    4 +
 .../carbondata/sdk/file/JsonCarbonWriter.java      |    3 +-
 .../org/apache/carbondata/sdk/file/RowUtil.java    |   11 +
 .../apache/carbondata/sdk/file/utils/SDKUtil.java  |   79 ++
 .../carbondata/sdk/file/CSVCarbonWriterTest.java   |   16 +-
 .../carbondata/sdk/file/CarbonReaderTest.java      |  186 +++-
 .../org/apache/carbondata/sdk/file/ImageTest.java  |  818 ++++++++++++++
 .../org/apache/carbondata/util/BinaryUtil.java     |  126 +++
 .../src/test/resources/image/carbondatalogo.jpg    |  Bin 0 -> 59099 bytes
 .../image/flowers/10686568196_b1915544a8.jpg       |  Bin 0 -> 97920 bytes
 .../image/flowers/10686568196_b1915544a8.txt       |    1 +
 .../image/flowers/10712722853_5632165b04.jpg       |  Bin 0 -> 63389 bytes
 .../image/flowers/10712722853_5632165b04.txt       |    1 +
 .../flowers/subfolder/10841136265_af473efc60.jpg   |  Bin 0 -> 62144 bytes
 .../flowers/subfolder/10841136265_af473efc60.txt   |    1 +
 .../src/test/resources/image/voc/2007_000027.jpg   |  Bin 0 -> 145493 bytes
 .../src/test/resources/image/voc/2007_000027.xml   |   63 ++
 .../src/test/resources/image/voc/2007_000032.jpg   |  Bin 0 -> 54757 bytes
 .../src/test/resources/image/voc/2007_000032.xml   |   63 ++
 .../src/test/resources/image/voc/2007_000033.jpg   |  Bin 0 -> 71205 bytes
 .../src/test/resources/image/voc/2007_000033.xml   |   51 +
 .../src/test/resources/image/voc/2007_000039.jpg   |  Bin 0 -> 64668 bytes
 .../src/test/resources/image/voc/2007_000039.xml   |   27 +
 .../src/test/resources/image/voc/2009_001444.jpg   |  Bin 0 -> 677151 bytes
 .../src/test/resources/image/voc/2009_001444.xml   |   28 +
 .../image/vocForSegmentationClass/2007_000032.jpg  |  Bin 0 -> 54757 bytes
 .../image/vocForSegmentationClass/2007_000032.png  |  Bin 0 -> 2334 bytes
 .../image/vocForSegmentationClass/2007_000033.jpg  |  Bin 0 -> 71205 bytes
 .../image/vocForSegmentationClass/2007_000033.png  |  Bin 0 -> 2814 bytes
 .../image/vocForSegmentationClass/2007_000042.jpg  |  Bin 0 -> 82847 bytes
 .../image/vocForSegmentationClass/2007_000042.png  |  Bin 0 -> 3620 bytes
 .../org/apache/carbondata/tool/CarbonCliTest.java  |   61 +-
 183 files changed, 7482 insertions(+), 916 deletions(-)
 create mode 100644 core/src/main/java/org/apache/carbondata/core/datamap/DataMapFilter.java
 copy core/src/main/java/org/apache/carbondata/core/metadata/datatype/{StringType.java =>
BinaryType.java} (81%)
 create mode 100644 core/src/main/java/org/apache/carbondata/core/scan/result/iterator/ColumnDriftRawResultIterator.java
 create mode 100644 integration/spark-common-test/src/test/java/org/apache/carbondata/sdk/util/BinaryUtil.java
 create mode 100644 integration/spark-common-test/src/test/resources/binaryStringNullData.csv
 create mode 100644 integration/spark-common-test/src/test/resources/binarydata.csv
 create mode 100644 integration/spark-common-test/src/test/resources/binarystringdata.csv
 create mode 100644 integration/spark-common-test/src/test/resources/binarystringdata2.csv
 create mode 100644 integration/spark-common-test/src/test/resources/decimalData.csv
 create mode 100644 integration/spark-common-test/src/test/resources/sort_columns/alldatatype1.csv
 create mode 100644 integration/spark-common-test/src/test/resources/sort_columns/alldatatype2.csv
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/binary/TestBinaryDataType.scala
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableSortColumnsProperty.scala
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableWithPageSizeInMb.scala
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTableForBinary.scala
 rename integration/{spark2 => spark-common}/src/main/scala/org/apache/spark/CarbonInputMetrics.scala
(100%)
 create mode 100644 integration/spark-datasource/src/test/java/org/apache/carbondata/sdk/util/BinaryUtil.java
 create mode 100644 integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceBinaryTest.scala
 copy processing/src/test/resources/schemas/modifiedTime.mdt => integration/spark2/src/test/scala/org/apache/spark/sql/CarbonGetTableDetailComandTestCase.scala
(100%)
 create mode 100644 processing/src/main/java/org/apache/carbondata/processing/loading/converter/impl/BinaryFieldConverterImpl.java
 create mode 100644 store/sdk/src/main/java/org/apache/carbondata/sdk/file/utils/SDKUtil.java
 create mode 100644 store/sdk/src/test/java/org/apache/carbondata/sdk/file/ImageTest.java
 create mode 100644 store/sdk/src/test/java/org/apache/carbondata/util/BinaryUtil.java
 create mode 100644 store/sdk/src/test/resources/image/carbondatalogo.jpg
 create mode 100644 store/sdk/src/test/resources/image/flowers/10686568196_b1915544a8.jpg
 create mode 100644 store/sdk/src/test/resources/image/flowers/10686568196_b1915544a8.txt
 create mode 100644 store/sdk/src/test/resources/image/flowers/10712722853_5632165b04.jpg
 create mode 100644 store/sdk/src/test/resources/image/flowers/10712722853_5632165b04.txt
 create mode 100644 store/sdk/src/test/resources/image/flowers/subfolder/10841136265_af473efc60.jpg
 create mode 100644 store/sdk/src/test/resources/image/flowers/subfolder/10841136265_af473efc60.txt
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000027.jpg
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000027.xml
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000032.jpg
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000032.xml
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000033.jpg
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000033.xml
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000039.jpg
 create mode 100755 store/sdk/src/test/resources/image/voc/2007_000039.xml
 create mode 100755 store/sdk/src/test/resources/image/voc/2009_001444.jpg
 create mode 100755 store/sdk/src/test/resources/image/voc/2009_001444.xml
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000032.jpg
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000032.png
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000033.jpg
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000033.png
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000042.jpg
 create mode 100755 store/sdk/src/test/resources/image/vocForSegmentationClass/2007_000042.png


Mime
View raw message