carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipes...@apache.org
Subject [carbondata] branch branch-1.5 updated (441edbb -> 6a57b4b)
Date Tue, 02 Apr 2019 02:41:20 GMT
This is an automated email from the ASF dual-hosted git repository.

ravipesala pushed a change to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/carbondata.git.


    from 441edbb  [maven-release-plugin] prepare for next development iteration
     new d55164f  [CARBONDATA-3287]Remove the validation for same schema in a location and
fix drop datamap issue
     new 3df5a2f  [CARBONDATA-3284] [CARBONDATA-3285] Workaround for Create-PreAgg Datamap
Fail & Sort-Columns Fix
     new 9e6b544  [CARBONDATA-3278] Remove duplicate code to get filter string of date/timestamp
     new 459ad23  [CARBONDATA-3107] Optimize error/exception coding for better debugging
     new 1c4c6dc  [CARBONDATA-2447] Block update operation on range/list/hash partition table
     new eef1eb1  [CARBONDATA-3299] Desc Formatted Issue Fixed
     new f6e1c2e  [CARBONDATA-3276] Compacting table that do not exist should modify the message
of MalformedCarbonCommandException
     new 02cc2b2  [CARBONDATA-3298]Removed Log Message for Already Deleted Segments
     new fdb48d0  [CARBONDATA-3305] Support show metacache command to list the cache sizes
for all tables
     new 4eefe52  [CARBONDATA-3305] Added DDL to drop cache for a table
     new ffa7730  [CARBONDATA-3281] Add validation for the size of the LRU cache
     new 6fa9bd2  [CARBONDATA-3301]Fix inserting null values to Array<date> columns
in carbon file format data load
     new 7e83df1  [LOG] Optimize the logs of CarbonProperties
     new 46fc6c5  [CARBONDATA-3307] Fix Performance Issue in No Sort
     new 9ed8184  [CARBONDATA-3297] Fix that the IndexoutOfBoundsException when creating table
and dropping table are at the same time
     new bfdff7f  [CARBONDATA-3300] Fixed ClassNotFoundException when using UDF in spark-shell
     new ded8885  [DOC] Update the doc of "Show DataMap"
     new 271fd55  [CARBONDATA-3304] Distinguish the thread names created by thread pool of
CarbonThreadFactory
     new 79d91fe  [CARBONDATA-3313] count(*) is not invalidating the invalid segments cache
     new d78eed4  [CARBONDATA-3315] Fix for Range Filter failing with two between clauses
as children of OR expression
     new 3f6a853  [CARBONDATA-3314] Fix for Index Cache Size in SHOW METACACHE DDL
     new 6975346  [CARBONDATA-3317] Fix NPE when execute 'show segments' command for stream
table
     new bfc912a  [CARBONDATA-3311] support presto 0.217 #3142
     new 4364473  [TestCase][HOTFIX] Added drop database in beforeEach to avoid exception
     new 15f13ad  [CARBONDATA-3302] [Spark-Integration] code cleaning related to CarbonCreateTable
command
     new c4f32dd  [CARBONDATA-3318] Added PreAgg & Bloom Event-Listener for ShowCacheCommmand
     new ef8001e  [CARBONDATA-3293] Prune datamaps improvement for count(*)
     new fbea5c6  [CARBONDATA-3321] Improved Single/Concurrent query Performance
     new 976e48a  [DOC] Fix the spell mistake of enable.unsafe.in.query.processing
     new 0f6ab06  [CARBONDATA-3322] [CARBONDATA-3323] Added check for invalid tables in ShowCacheCommand
& Standard output on ShowCacheCommand on table
     new b081014  [CARBONDATA-3320]fix number of partitions issue in describe formatted and
drop partition issue
     new 236b5e1  [CARBONDATA-3329] Fixed deadlock issue during failed query
     new f4141cb  [CARBONDATA-3328]Fixed performance issue with merge small files distribution
     new cde80f2  [HOTFIX][DOC] Optimize quick-start-guide.md and dml-of-carbondata.md
     new e565d1f  [CARBONDATA-3319][TestCase]Added condition to check if datamap exist or
not before caching
     new 09c598f  [CARBONDATA-3330] Fix Invalid Exception while clearing datamap from SDK
carbon reader
     new d52ef32  [CARBONDATA-3333]Fixed No Sort Store Size issue and Compatibility issue
after alter added column done in 1.1 and load in 1.5
     new 8ae260f  [Document] update doc about presto version support details #3163
     new 81e9714  [CARBONDATA-3332] Blocked concurrent compaction and update/delete
     new 6a2c072  [CARBONDATA-3335]Fixed load and compaction failure after alter done in older
version
     new 6a57b4b  [HOTFIX]Fixed data map loading issue when number of segments are high

The 41 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../carbondata/core/cache/CacheProvider.java       |   4 +
 .../carbondata/core/cache/CarbonLRUCache.java      |  59 +++-
 .../cache/dictionary/ForwardDictionaryCache.java   |   2 +-
 .../cache/dictionary/ReverseDictionaryCache.java   |   2 +-
 .../core/constants/CarbonCommonConstants.java      |   5 +
 .../constants/CarbonCommonConstantsInternal.java   |   2 +
 .../core/datamap/DataMapStoreManager.java          |  39 ++-
 .../carbondata/core/datamap/DataMapUtil.java       |   2 +-
 .../core/datamap/DistributableDataMapFormat.java   |   1 +
 .../apache/carbondata/core/datamap/Segment.java    |  24 +-
 .../carbondata/core/datamap/TableDataMap.java      | 118 +++++--
 .../carbondata/core/datamap/dev/DataMap.java       |  19 +-
 .../datamap/dev/cgdatamap/CoarseGrainDataMap.java  |  18 +-
 .../datamap/dev/fgdatamap/FineGrainDataMap.java    |  17 +-
 .../carbondata/core/datastore/TableSpec.java       |  85 ++++-
 .../block/SegmentPropertiesAndSchemaHolder.java    |  62 ++--
 .../UnsafeAbstractDimensionDataChunkStore.java     |  10 +-
 .../filesystem/AbstractDFSCarbonFile.java          |  14 +-
 .../datastore/filesystem/AlluxioCarbonFile.java    |   2 +-
 .../core/datastore/filesystem/HDFSCarbonFile.java  |   2 +-
 .../core/datastore/filesystem/LocalCarbonFile.java |   2 +-
 .../core/datastore/filesystem/S3CarbonFile.java    |   2 +-
 .../datastore/filesystem/ViewFSCarbonFile.java     |   2 +-
 .../core/datastore/impl/FileFactory.java           |  47 ++-
 .../client/NonSecureDictionaryClient.java          |   2 +-
 .../client/NonSecureDictionaryClientHandler.java   |   4 +-
 .../generator/TableDictionaryGenerator.java        |   2 +-
 .../server/NonSecureDictionaryServerHandler.java   |   2 +-
 .../service/AbstractDictionaryServer.java          |   8 +-
 .../core/indexstore/BlockletDataMapIndexStore.java |   4 +-
 .../core/indexstore/BlockletDetailInfo.java        |   2 +-
 .../core/indexstore/ExtendedBlocklet.java          |  97 ++++--
 .../core/indexstore/SegmentPropertiesFetcher.java  |   3 +
 .../TableBlockIndexUniqueIdentifier.java           |   5 +-
 .../core/indexstore/UnsafeMemoryDMStore.java       | 161 ++++++---
 .../indexstore/blockletindex/BlockDataMap.java     | 157 +++++----
 .../indexstore/blockletindex/BlockletDataMap.java  |  28 +-
 .../blockletindex/BlockletDataMapFactory.java      |   9 +-
 .../blockletindex/BlockletDataMapRowIndexes.java   |  14 +-
 .../carbondata/core/indexstore/row/DataMapRow.java |  12 +-
 .../core/indexstore/row/UnsafeDataMapRow.java      | 217 ++-----------
 .../core/indexstore/schema/CarbonRowSchema.java    |   8 +
 .../core/indexstore/schema/SchemaGenerator.java    |  72 ++++
 .../timestamp/DateDirectDictionaryGenerator.java   |   4 +-
 .../apache/carbondata/core/locks/LockUsage.java    |   1 +
 .../carbondata/core/locks/ZookeeperInit.java       |   2 +-
 .../core/memory/UnsafeMemoryManager.java           |   3 +-
 .../carbondata/core/metadata/SegmentFileStore.java |  21 +-
 .../core/metadata/blocklet/BlockletInfo.java       |   4 +-
 .../core/metadata/schema/table/CarbonTable.java    |  29 +-
 .../carbondata/core/mutate/CarbonUpdateUtil.java   |  14 +-
 .../core/reader/CarbonDeleteFilesDataReader.java   |  14 +-
 .../scan/executor/impl/AbstractQueryExecutor.java  |  22 +-
 .../scan/expression/RangeExpressionEvaluator.java  |  30 +-
 .../carbondata/core/scan/filter/FilterUtil.java    |  16 +-
 .../carbondata/core/scan/model/QueryModel.java     |  64 ++--
 .../core/scan/result/BlockletScannedResult.java    |   6 +-
 .../AbstractDetailQueryResultIterator.java         |   2 +-
 .../core/statusmanager/LoadMetadataDetails.java    |   6 +-
 .../core/statusmanager/SegmentStatusManager.java   |   6 +-
 .../carbondata/core/util/BlockletDataMapUtil.java  |  33 +-
 .../carbondata/core/util/CarbonProperties.java     | 134 ++++----
 .../apache/carbondata/core/util/CarbonUtil.java    |  46 ++-
 .../apache/carbondata/core/util/DataTypeUtil.java  |   8 +-
 .../carbondata/core/util/DeleteLoadFolders.java    |   4 -
 .../core/util/ObjectSerializationUtil.java         |   7 +-
 .../carbondata/core/util/path/HDFSLeaseUtils.java  |   2 +-
 .../apache/carbondata/hadoop/CarbonInputSplit.java | 361 +++++++++++++++------
 .../hadoop/internal/ObjectArrayWritable.java       |   0
 .../carbondata/hadoop/internal/index/Block.java    |   0
 .../carbondata/core/cache/CarbonLRUCacheTest.java  |   7 +
 .../datastore/filesystem/HDFSCarbonFileTest.java   |   2 +-
 .../core/load/LoadMetadataDetailsUnitTest.java     |   2 +-
 .../datamap/bloom/BloomCacheKeyValue.java          |   2 +-
 .../datamap/bloom/BloomCoarseGrainDataMap.java     |  41 +--
 .../bloom/BloomCoarseGrainDataMapFactory.java      |  11 +-
 .../datamap/lucene/LuceneFineGrainDataMap.java     |   6 +-
 .../lucene/LuceneFineGrainDataMapFactory.java      |   2 +-
 docs/datamap/datamap-management.md                 |   1 +
 docs/ddl-of-carbondata.md                          |  43 +++
 docs/dml-of-carbondata.md                          |  24 +-
 docs/faq.md                                        |  18 +
 docs/presto-guide.md                               |  33 +-
 docs/quick-start-guide.md                          |   6 +-
 docs/usecases.md                                   |   4 +-
 .../carbondata/hadoop/CarbonMultiBlockSplit.java   |  23 +-
 .../carbondata/hadoop/CarbonRecordReader.java      |   4 +-
 .../hadoop/api/CarbonFileInputFormat.java          |   6 +-
 .../carbondata/hadoop/api/CarbonInputFormat.java   |  55 +---
 .../hadoop/api/CarbonTableInputFormat.java         | 117 +++----
 .../hadoop/api/CarbonTableOutputFormat.java        |   3 +-
 .../hadoop/util/CarbonVectorizedRecordReader.java  |   2 +-
 .../hive/CarbonDictionaryDecodeReadSupport.java    |   2 +-
 .../carbondata/hive/MapredCarbonInputFormat.java   |   2 +-
 integration/presto/pom.xml                         |   2 +-
 .../presto/CarbondataConnectorFactory.java         |   7 +-
 .../apache/carbondata/presto/CarbondataModule.java |   8 +-
 .../carbondata/presto/CarbondataSplitManager.java  |   2 +-
 .../presto/impl/CarbonLocalInputSplit.java         |   4 +-
 .../carbondata/presto/impl/CarbonTableReader.java  |  12 +-
 .../PrestoTestNonTransactionalTableFiles.scala     |   2 +-
 .../cluster/sdv/generated/AlterTableTestCase.scala |  10 +-
 .../cluster/sdv/generated/QueriesBVATestCase.scala |   4 +-
 ...teTableUsingSparkCarbonFileFormatTestCase.scala |   7 +-
 .../datasource/SparkCarbonDataSourceTestCase.scala |  19 +-
 .../carbondata/cluster/sdv/suite/SDVSuites.scala   |   2 +-
 .../sql/common/util/DataSourceTestUtil.scala}      |  88 ++---
 ...ryWithColumnMetCacheAndCacheLevelProperty.scala |   5 +-
 .../dblocation/DBLocationCarbonTableTestCase.scala |  26 +-
 .../detailquery/RangeFilterTestCase.scala          |  38 +++
 .../TestAllDataTypeForPartitionTable.scala         |   4 +-
 .../partition/TestDDLForPartitionTable.scala       |  13 +-
 .../partition/TestUpdateForPartitionTable.scala    |  71 ++++
 ...StandardPartitionWithPreaggregateTestCase.scala |  22 ++
 .../sql/commands/TestCarbonDropCacheCommand.scala  | 200 ++++++++++++
 .../sql/commands/TestCarbonShowCacheCommand.scala  | 233 +++++++++++++
 .../client/SecureDictionaryClientHandler.java      |   4 +-
 .../server/SecureDictionaryServerHandler.java      |   2 +-
 .../org/apache/carbondata/spark/util/Util.java     |   3 +-
 .../org/apache/carbondata/api/CarbonStore.scala    |  10 +-
 .../{CleanFilesEvents.scala => CacheEvents.scala}  |  25 +-
 .../org/apache/carbondata/events/Events.scala      |  14 +
 .../carbondata/spark/rdd/CarbonMergerRDD.scala     |   4 +-
 .../carbondata/spark/rdd/CarbonScanRDD.scala       |  13 +-
 .../apache/carbondata/spark/util/CommonUtil.scala  |  39 ++-
 .../spark/sql/catalyst/CarbonDDLSqlParser.scala    |   1 +
 .../vectorreader/VectorizedCarbonRecordReader.java |   4 +-
 .../datasources/SparkCarbonFileFormat.scala        |   6 +-
 .../datasource/SparkCarbonDataSourceTest.scala     |  19 +-
 .../spark/rdd/CarbonDataRDDFactory.scala           |  48 +--
 .../org/apache/spark/sql/CarbonCountStar.scala     |   2 +-
 .../spark/sql/CarbonDatasourceHadoopRelation.scala |   6 +-
 .../scala/org/apache/spark/sql/CarbonEnv.scala     |   5 +
 .../sql/execution/command/cache/CacheUtil.scala    | 114 +++++++
 .../command/cache/CarbonDropCacheCommand.scala     |  66 ++++
 .../command/cache/CarbonShowCacheCommand.scala     | 225 +++++++++++++
 .../command/cache/DropCacheEventListeners.scala    | 121 +++++++
 .../command/cache/ShowCacheEventListeners.scala    | 126 +++++++
 .../CarbonAlterTableCompactionCommand.scala        |  55 ++--
 .../command/management/CarbonLoadDataCommand.scala |   2 +-
 .../mutation/CarbonProjectForDeleteCommand.scala   |  18 +-
 .../mutation/CarbonProjectForUpdateCommand.scala   |  99 +++---
 .../command/mutation/DeleteExecution.scala         |   2 +-
 .../schema/CarbonAlterTableRenameCommand.scala     |   2 -
 .../command/table/CarbonCreateTableCommand.scala   |   3 +-
 .../table/CarbonDescribeFormattedCommand.scala     |  39 ++-
 .../command/table/CarbonDropTableCommand.scala     |  32 +-
 .../spark/sql/execution/strategy/DDLStrategy.scala |  12 +-
 .../spark/sql/hive/CarbonFileMetastore.scala       |  37 ++-
 .../spark/sql/parser/CarbonSpark2SqlParser.scala   |  18 +-
 .../BloomCoarseGrainDataMapFunctionSuite.scala     |   2 +-
 .../register/TestRegisterCarbonTable.scala         |  24 +-
 .../processing/datamap/DataMapWriterListener.java  |   2 +-
 .../processing/datatypes/PrimitiveDataType.java    |   7 +-
 .../loading/AbstractDataLoadProcessorStep.java     |   2 +-
 .../loading/TableProcessingOperations.java         |   3 +-
 .../loading/converter/impl/RowConverterImpl.java   |   4 +-
 .../loading/model/CarbonLoadModelBuilder.java      |   2 +-
 .../loading/parser/impl/JsonRowParser.java         |   2 +-
 .../loading/row/IntermediateSortTempRow.java       |   8 +
 .../loading/sort/SortStepRowHandler.java           |   7 +-
 .../sort/impl/ParallelReadMergeSorterImpl.java     |   5 +-
 ...ParallelReadMergeSorterWithColumnRangeImpl.java |   2 +-
 .../UnsafeBatchParallelReadMergeSorterImpl.java    |   4 +-
 .../impl/UnsafeParallelReadMergeSorterImpl.java    |   5 +-
 ...ParallelReadMergeSorterWithColumnRangeImpl.java |   2 +-
 .../loading/sort/unsafe/UnsafeCarbonRowPage.java   |   8 +-
 .../loading/sort/unsafe/UnsafeSortDataRows.java    |  23 +-
 .../holder/UnsafeSortTempFileChunkHolder.java      |  26 +-
 .../merger/UnsafeIntermediateFileMerger.java       |   2 +-
 .../unsafe/merger/UnsafeIntermediateMerger.java    |   7 +-
 .../UnsafeSingleThreadFinalSortFilesMerger.java    |   4 +-
 .../CarbonRowDataWriterProcessorStepImpl.java      |  10 +-
 .../steps/DataWriterBatchProcessorStepImpl.java    |   4 +-
 .../loading/steps/DataWriterProcessorStepImpl.java |   2 +-
 .../loading/steps/InputProcessorStepImpl.java      |   2 +-
 .../processing/merger/CarbonCompactionUtil.java    |   8 +-
 .../processing/merger/CarbonDataMergerUtil.java    |   4 +-
 .../merger/CompactionResultSortProcessor.java      |  24 +-
 .../merger/RowResultMergerProcessor.java           |   2 +-
 .../partition/spliter/RowResultProcessor.java      |   6 +-
 .../DummyRowUpdater.java}                          |  25 +-
 .../processing/sort/SchemaBasedRowUpdater.java     |  91 ++++++
 .../SortTempRowUpdater.java}                       |  27 +-
 .../sortdata/SingleThreadFinalSortFilesMerger.java |   4 +-
 .../processing/sort/sortdata/SortDataRows.java     |   5 +-
 .../sort/sortdata/SortIntermediateFileMerger.java  |   3 +-
 .../processing/sort/sortdata/SortParameters.java   |  54 +++
 .../sort/sortdata/SortTempFileChunkHolder.java     |  28 +-
 .../processing/sort/sortdata/TableFieldStat.java   |  16 +
 .../store/CarbonFactDataHandlerColumnar.java       |  12 +-
 .../store/writer/AbstractFactDataWriter.java       |   6 +-
 .../processing/util/CarbonLoaderUtil.java          |   6 +-
 .../apache/carbondata/store/LocalCarbonStore.java  |   4 +-
 .../carbondata/sdk/file/CarbonReaderTest.java      |  38 +--
 .../java/org/apache/carbondata/tool/CarbonCli.java |   4 +-
 196 files changed, 3663 insertions(+), 1430 deletions(-)
 rename {hadoop => core}/src/main/java/org/apache/carbondata/hadoop/CarbonInputSplit.java
(56%)
 rename {hadoop => core}/src/main/java/org/apache/carbondata/hadoop/internal/ObjectArrayWritable.java
(100%)
 rename {hadoop => core}/src/main/java/org/apache/carbondata/hadoop/internal/index/Block.java
(100%)
 copy integration/{spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/TestUtil.scala
=> spark-common-cluster-test/src/test/scala/org/apache/spark/sql/common/util/DataSourceTestUtil.scala}
(67%)
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/partition/TestUpdateForPartitionTable.scala
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonDropCacheCommand.scala
 create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonShowCacheCommand.scala
 copy integration/spark-common/src/main/scala/org/apache/carbondata/events/{CleanFilesEvents.scala
=> CacheEvents.scala} (72%)
 create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CacheUtil.scala
 create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
 create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
 create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/DropCacheEventListeners.scala
 create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/ShowCacheEventListeners.scala
 copy processing/src/main/java/org/apache/carbondata/processing/{loading/sort/unsafe/holder/SortTempChunkHolder.java
=> sort/DummyRowUpdater.java} (54%)
 create mode 100644 processing/src/main/java/org/apache/carbondata/processing/sort/SchemaBasedRowUpdater.java
 copy processing/src/main/java/org/apache/carbondata/processing/{loading/sort/unsafe/holder/SortTempChunkHolder.java
=> sort/SortTempRowUpdater.java} (53%)


Mime
View raw message