carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From qiang...@apache.org
Subject [1/2] carbondata git commit: Streaming Ingestion
Date Tue, 30 Jan 2018 04:04:25 GMT
Repository: carbondata
Updated Branches:
  refs/heads/master ab763474f -> 9ec64a6d2


Streaming Ingestion


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/b73a221a
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/b73a221a
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/b73a221a

Branch: refs/heads/master
Commit: b73a221a16fcbf0b14ad492e3ffc17902b15d4cd
Parents: ab76347
Author: sgururajshetty <sgururajshetty@gmail.com>
Authored: Mon Jan 29 19:34:14 2018 +0530
Committer: sgururajshetty <sgururajshetty@gmail.com>
Committed: Mon Jan 29 19:34:14 2018 +0530

----------------------------------------------------------------------
 docs/configuration-parameters.md      | 2 ++
 docs/data-management-on-carbondata.md | 8 ++++++++
 2 files changed, 10 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/b73a221a/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index fc10646..5794f4d 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -36,6 +36,8 @@ This section provides the details of all the configurations required for
the Car
 | carbon.ddl.base.hdfs.url | hdfs://hacluster/opt/data | This property is used to configure
the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to
the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass
the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv,
the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure
the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path
as /2016/xyz.csv. |
 | carbon.badRecords.location | /opt/Carbon/Spark/badrecords | Path where the bad records
are stored. |
 | carbon.data.file.version | 2 | If this parameter value is set to 1, then CarbonData will
support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards
version), then CarbonData will support the data load of new format only.|
+| carbon.streaming.auto.handoff.enabled | true | If this parameter value is set to true,
auto trigger handoff function will be enabled.|
+| carbon.streaming.segment.max.size | 1024000000 | This parameter defines the maximum size
of the streaming segment. Setting this parameter to appropriate value will avoid impacting
the streaming ingestion. The value is in bytes.|
 
 ##  Performance Configuration
 This section provides the details of all the configurations required for CarbonData Performance
Optimization.

http://git-wip-us.apache.org/repos/asf/carbondata/blob/b73a221a/docs/data-management-on-carbondata.md
----------------------------------------------------------------------
diff --git a/docs/data-management-on-carbondata.md b/docs/data-management-on-carbondata.md
index a4fb40b..d1415ff 100644
--- a/docs/data-management-on-carbondata.md
+++ b/docs/data-management-on-carbondata.md
@@ -110,7 +110,14 @@ This tutorial is going to introduce all commands and data operations
on CarbonDa
                     'COMPACTION_PRESERVE_SEGMENTS'='10',
                     'ALLOWED_COMPACTION_DAYS'='5')
      ```
+     
+   - **Streaming**
+
+     CarbonData supports streaming ingestion for real-time data. You can create the ‘streaming’
table using the following table properties.
 
+     ```
+     TBLPROPERTIES (‘streaming’='true’)
+     ```
 
 ### Example:
 
@@ -134,6 +141,7 @@ This tutorial is going to introduce all commands and data operations on
CarbonDa
                    'AUTO_LOAD_MERGE'='true',
                    'COMPACTION_LEVEL_THRESHOLD'='5,6',
                    'COMPACTION_PRESERVE_SEGMENTS'='10',
+				   ‘streaming’='true’,
                    'ALLOWED_COMPACTION_DAYS'='5')
    ```
         


Mime
View raw message