dolphinscheduler-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lidong...@apache.org
Subject [incubator-dolphinscheduler-website] branch master updated: Added English page for plugin development page, migrated developer guide to developer, removed blog system architecture design page and style modification
Date Fri, 29 Nov 2019 11:57:09 GMT
This is an automated email from the ASF dual-hosted git repository.

lidongdai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-dolphinscheduler-website.git


The following commit(s) were added to refs/heads/master by this push:
     new e7bbc7f  Added English page for plugin development page, migrated developer guide
to developer, removed blog system architecture design page and style modification
     new 4bf11bd  Merge pull request #42 from break60/master
e7bbc7f is described below

commit e7bbc7fe2f510767941c34e277206f60934d3913
Author: break60 <790061044@qq.com>
AuthorDate: Fri Nov 29 17:32:11 2019 +0800

    Added English page for plugin development page, migrated developer guide to developer,
removed blog system architecture design page and style modification
---
 .../architecture-design.md                         |  0
 .../backend-development.md                         |  0
 .../frontend-development.md                        |  0
 docs/en-us/development/plugin-development.md       | 54 +++++++++++++++
 .../architecture-design.md                         | 78 ++++++++++++++++++++++
 .../backend-development.md                         |  0
 .../frontend-development.md                        |  0
 .../plugin-development.md                          |  0
 md_json/docs.json                                  | 39 ++++++-----
 site_config/blog.js                                | 14 ----
 site_config/development.js                         | 52 +++++++++++++++
 site_config/docs.js                                | 48 -------------
 site_config/site.js                                |  4 +-
 src/components/header/index.jsx                    | 20 +++---
 src/pages/blogDetail/index.scss                    |  2 +-
 15 files changed, 217 insertions(+), 94 deletions(-)

diff --git a/docs/en-us/developer_guide/architecture-design.md b/docs/en-us/development/architecture-design.md
similarity index 100%
rename from docs/en-us/developer_guide/architecture-design.md
rename to docs/en-us/development/architecture-design.md
diff --git a/docs/en-us/developer_guide/backend-development.md b/docs/en-us/development/backend-development.md
similarity index 100%
rename from docs/en-us/developer_guide/backend-development.md
rename to docs/en-us/development/backend-development.md
diff --git a/docs/en-us/developer_guide/frontend-development.md b/docs/en-us/development/frontend-development.md
similarity index 100%
rename from docs/en-us/developer_guide/frontend-development.md
rename to docs/en-us/development/frontend-development.md
diff --git a/docs/en-us/development/plugin-development.md b/docs/en-us/development/plugin-development.md
new file mode 100644
index 0000000..eda2d82
--- /dev/null
+++ b/docs/en-us/development/plugin-development.md
@@ -0,0 +1,54 @@
+## Task Plugin Development
+
+Remind:Currently, task plugin development does not support hot deployment.
+
+### Shell-based tasks
+
+#### YARN-based calculations (see MapReduceTask)
+
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** Create a custom
task in the class (also need to register the corresponding task type in TaskType)
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractYarnTask**
+- Constructor Scheduling **AbstractYarnTask** Construction method
+- Inherit **AbstractParameters** Custom task parameter entity
+- Rewrite **AbstractTask** of **init** Parsing in method**Custom task parameters**
+- Rewrite **buildCommand** Encapsulation command
+
+
+
+#### Non-YARN-based calculations (see ShellTask)
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** A custom task
+
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractTask**
+
+- Instantiation in constructor **ShellCommandExecutor**
+
+  ```
+  public ShellTask(TaskProps props, Logger logger) {
+    super(props, logger);
+  
+    this.taskDir = props.getTaskDir();
+  
+    this.processTask = new ShellCommandExecutor(this::logHandle,
+        props.getTaskDir(), props.getTaskAppId(),
+        props.getTenantCode(), props.getEnvFile(), props.getTaskStartTime(),
+        props.getTaskTimeout(), logger);
+    this.processDao = DaoFactory.getDaoInstance(ProcessDao.class);
+  }
+  ```
+
+  Incoming custom tasks **TaskProps**And custom**Logger**,TaskProps Encapsulate task information,
Logger is installed with custom log information
+
+- Inherit **AbstractParameters** Custom task parameter entity
+
+- Rewrite **AbstractTask** of **init** Parsing in method**Custom task parameter entity**
+
+- Rewrite **handle** method,transfer **ShellCommandExecutor** of **run** method,The first
parameter is passed in**command**,Pass the second parameter to ProcessDao and set the corresponding
**exitStatusCode**
+
+### Non-SHELL-based tasks (see SqlTask)
+
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** A custom task
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractTask**
+- Inherit **AbstractParameters** Custom task parameter entity
+- Constructor or override **AbstractTask** of **init** in the method, parse the custom task
parameter entity
+- Rewrite **handle** Methods to implement business logic and set the corresponding**exitStatusCode**
+
diff --git a/docs/zh-cn/developer_guide/architecture-design.md b/docs/zh-cn/development/architecture-design.md
similarity index 85%
rename from docs/zh-cn/developer_guide/architecture-design.md
rename to docs/zh-cn/development/architecture-design.md
index 087e834..2044f87 100755
--- a/docs/zh-cn/developer_guide/architecture-design.md
+++ b/docs/zh-cn/development/architecture-design.md
@@ -302,3 +302,81 @@ public class TaskLogFilter extends Filter<ILoggingEvent {
 本文从调度出发,初步介绍了大数据分布式工作流调度系统--DolphinScheduler的架构原理及实现思路。未完待续
 
 
+ some processes need to be processed before other processes. This can be configured at the
start of the process or at the time of scheduled start. There are 5 levels, followed by HIGHEST,
HIGH, MEDIUM, LOW, and LOWEST. As shown below
+
+      <p align="center">
+         <img src="https://analysys.github.io/easyscheduler_docs_cn/images/process_priority.png"
alt="Process Priority Configuration" width="40%" />
+       </p>
+
+    - The priority of the task is also divided into 5 levels, followed by HIGHEST, HIGH,
MEDIUM, LOW, and LOWEST. As shown below
+
+      <p align="center">
+         <img src="https://analysys.github.io/easyscheduler_docs_cn/images/task_priority.png"
alt="task priority configuration" width="35%" />
+       </p>
+
+##### VI. Logback and gRPC implement log access
+
+- Since the Web (UI) and Worker are not necessarily on the same machine, viewing the log
is not as it is for querying local files. There are two options:
+  - Put the logs on the ES search engine
+  - Obtain remote log information through gRPC communication
+- Considering the lightweightness of DolphinScheduler as much as possible, gRPC was chosen
to implement remote access log information.
+
+ <p align="center">
+   <img src="https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png" alt="grpc
remote access" width="50%" />
+ </p>
+
+- We use a custom Logback FileAppender and Filter function to generate a log file for each
task instance.
+- The main implementation of FileAppender is as follows:
+
+```java
+ /**
+  * task log appender
+  */
+ Public class TaskLogAppender extends FileAppender<ILoggingEvent {
+ 
+     ...
+
+    @Override
+    Protected void append(ILoggingEvent event) {
+
+        If (currentlyActiveFile == null){
+            currentlyActiveFile = getFile();
+        }
+        String activeFile = currentlyActiveFile;
+        // thread name: taskThreadName-processDefineId_processInstanceId_taskInstanceId
+        String threadName = event.getThreadName();
+        String[] threadNameArr = threadName.split("-");
+        // logId = processDefineId_processInstanceId_taskInstanceId
+        String logId = threadNameArr[1];
+        ...
+        super.subAppend(event);
+    }
+}
+```
+
+Generate a log in the form of /process definition id/process instance id/task instance id.log
+
+- Filter matches the thread name starting with TaskLogInfo:
+- TaskLogFilter is implemented as follows:
+
+```java
+ /**
+ * task log filter
+ */
+Public class TaskLogFilter extends Filter<ILoggingEvent {
+
+    @Override
+    Public FilterReply decide(ILoggingEvent event) {
+        If (event.getThreadName().startsWith("TaskLogInfo-")){
+            Return FilterReply.ACCEPT;
+        }
+        Return FilterReply.DENY;
+    }
+}
+```
+
+
+
+### summary
+
+Starting from the scheduling, this paper introduces the architecture principle and implementation
ideas of the big data distributed workflow scheduling system-DolphinScheduler. To be continued
\ No newline at end of file
diff --git a/docs/zh-cn/developer_guide/backend-development.md b/docs/zh-cn/development/backend-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/backend-development.md
rename to docs/zh-cn/development/backend-development.md
diff --git a/docs/zh-cn/developer_guide/frontend-development.md b/docs/zh-cn/development/frontend-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/frontend-development.md
rename to docs/zh-cn/development/frontend-development.md
diff --git a/docs/zh-cn/developer_guide/plugin-development.md b/docs/zh-cn/development/plugin-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/plugin-development.md
rename to docs/zh-cn/development/plugin-development.md
diff --git a/md_json/docs.json b/md_json/docs.json
index d770ea8..0b8a72f 100755
--- a/md_json/docs.json
+++ b/md_json/docs.json
@@ -2,17 +2,12 @@
   "en-us": [
     {
       "filename": "architecture-design.md",
-      "link": "/en-us/docs/developer_guide/architecture-design.html",
+      "link": "/en-us/docs/development/architecture-design.html",
       "meta": {}
     },
     {
       "filename": "backend-development.md",
-      "link": "/en-us/docs/developer_guide/backend-development.html",
-      "meta": {}
-    },
-    {
-      "filename": "frontend-development.md",
-      "link": "/en-us/docs/developer_guide/frontend-development.html",
+      "link": "/en-us/docs/development/backend-development.html",
       "meta": {}
     },
     {
@@ -26,6 +21,16 @@
       "meta": {}
     },
     {
+      "filename": "frontend-development.md",
+      "link": "/en-us/docs/development/frontend-development.html",
+      "meta": {}
+    },
+    {
+      "filename": "plugin-development.md",
+      "link": "/en-us/docs/development/plugin-development.html",
+      "meta": {}
+    },
+    {
       "filename": "faq.md",
       "link": "/en-us/docs/faq.html",
       "meta": {}
@@ -59,32 +64,32 @@
   "zh-cn": [
     {
       "filename": "architecture-design.md",
-      "link": "/zh-cn/docs/developer_guide/architecture-design.html",
+      "link": "/zh-cn/docs/development/architecture-design.html",
       "meta": {}
     },
     {
       "filename": "backend-development.md",
-      "link": "/zh-cn/docs/developer_guide/backend-development.html",
+      "link": "/zh-cn/docs/development/backend-development.html",
       "meta": {}
     },
     {
-      "filename": "frontend-development.md",
-      "link": "/zh-cn/docs/developer_guide/frontend-development.html",
+      "filename": "contribute.md",
+      "link": "/zh-cn/docs/development/contribute.html",
       "meta": {}
     },
     {
-      "filename": "plugin-development.md",
-      "link": "/zh-cn/docs/developer_guide/plugin-development.html",
+      "filename": "developers.md",
+      "link": "/zh-cn/docs/development/developers.html",
       "meta": {}
     },
     {
-      "filename": "contribute.md",
-      "link": "/zh-cn/docs/development/contribute.html",
+      "filename": "frontend-development.md",
+      "link": "/zh-cn/docs/development/frontend-development.html",
       "meta": {}
     },
     {
-      "filename": "developers.md",
-      "link": "/zh-cn/docs/development/developers.html",
+      "filename": "plugin-development.md",
+      "link": "/zh-cn/docs/development/plugin-development.html",
       "meta": {}
     },
     {
diff --git a/site_config/blog.js b/site_config/blog.js
index 50471cc..d85128c 100755
--- a/site_config/blog.js
+++ b/site_config/blog.js
@@ -9,13 +9,6 @@ export default {
         dateStr: '2019-9-27',
         desc: 'Apache Dolphin Scheduler(Incubating) will organize a meetup in Shanghai  2019.10.26.
Welcome to rigister.',
         link: '/en-us/blog/meetup_2019_10_26.html',
-      },
-      {
-        title: 'DolphinScheduler Architecture Desgin',
-        author: 'xiaochun',
-        dateStr: '2019-9-20',
-        desc: 'Blog description, some text to sum up the main content of the blog',
-        link: '/en-us/blog/architecture-design.html'
       }
     ],
   },
@@ -29,13 +22,6 @@ export default {
         dateStr: '2019-9-27',
         desc: 'Apache Dolphin Scheduler(Incubating) 将于2019年10月26日在上海组织一次会议,欢迎踊跃报名。
',
         link: '/zh-cn/blog/meetup_2019_10_26.html',
-      },
-      {
-        title: 'DolphinScheduler系统架构设计',
-        author: 'xiaochun.liu',
-        dateStr: '2019-9-20',
-        desc: '介绍DolphinScheduler整体架构',
-        link: '/zh-cn/blog/architecture-design.html',
       }
     ],
   },
diff --git a/site_config/development.js b/site_config/development.js
index 53ae908..ff0d3fa 100755
--- a/site_config/development.js
+++ b/site_config/development.js
@@ -2,6 +2,32 @@ export default {
   'en-us': {
     sidemenu: [
       {
+        title: 'Developer guide',
+        children: [
+          {
+            title: 'Architecture Design',
+            link: '/en-us/docs/development/architecture-design.html',
+          },
+          {
+            title: 'Backend Development',
+            link: '/en-us/docs/development/backend-development.html',
+          },
+          {
+            title: 'plugin-development',
+            link: '/en-us/docs/development/plugin-development.html',
+          },
+          {
+            title: 'Frontend Development',
+            link: '/en-us/docs/development/frontend-development.html',
+          },
+          {
+            title: 'Api Document',
+            target: '_blank',
+            link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=en&lang=en'
+          },
+        ],
+      },
+      {
         title: 'Committer Guide',
         children: [
           {
@@ -20,6 +46,32 @@ export default {
   'zh-cn': {
       sidemenu: [
         {
+          title: '开发者指南',
+          children: [
+            {
+              title: '架构设计',
+              link: '/zh-cn/docs/development/architecture-design.html',
+            },
+            {
+              title: '后端开发',
+              link: '/zh-cn/docs/development/backend-development.html',
+            },
+            {
+              title: '插件开发',
+              link: '/zh-cn/docs/development/plugin-development.html',
+            },
+            {
+              title: '前端开发',
+              link: '/zh-cn/docs/development/frontend-development.html',
+            },
+            {
+              title: '接口文档',
+              target: '_blank',
+              link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=zh_CN&lang=zh'
+            },
+          ],
+        },
+        {
           title: '提交者向导',
           children: [
             {
diff --git a/site_config/docs.js b/site_config/docs.js
index d0fb3ef..3a3c338 100755
--- a/site_config/docs.js
+++ b/site_config/docs.js
@@ -23,28 +23,6 @@ export default {
         ]
       },
       {
-        title: 'Developer guide',
-        children: [
-          {
-            title: 'Architecture Design',
-            link: '/en-us/docs/developer_guide/architecture-design.html',
-          },
-          {
-            title: 'Backend Development',
-            link: '/en-us/docs/developer_guide/backend-development.html',
-          },
-          {
-            title: 'Frontend Development',
-            link: '/en-us/docs/developer_guide/frontend-development.html',
-          },
-          {
-            title: 'Api Document',
-            target: '_blank',
-            link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=en&lang=en'
-          },
-        ],
-      },
-      {
         title: 'Release',
         children: [
           {
@@ -89,32 +67,6 @@ export default {
         ],
       },
       {
-        title: '开发者指南',
-        children: [
-          {
-            title: '架构设计',
-            link: '/zh-cn/docs/developer_guide/architecture-design.html',
-          },
-          {
-            title: '后端开发',
-            link: '/zh-cn/docs/developer_guide/backend-development.html',
-          },
-          {
-            title: '插件开发',
-            link: '/zh-cn/docs/developer_guide/architecture-design.html',
-          },
-          {
-            title: '前端开发',
-            link: '/zh-cn/docs/developer_guide/frontend-development.html',
-          },
-          {
-            title: '接口文档',
-            target: '_blank',
-            link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=zh_CN&lang=zh'
-          },
-        ],
-      },
-      {
         title: '版本发布',
         children: [
           {
diff --git a/site_config/site.js b/site_config/site.js
index 38c8b70..2c28e40 100755
--- a/site_config/site.js
+++ b/site_config/site.js
@@ -37,7 +37,7 @@ export default {
       {
         key: 'development',
         text: 'DEVELOPMENT',
-        link: '/en-us/docs/development/developers.html',
+        link: '/en-us/docs/development/architecture-design.html',
       },
       {
         key: 'community',
@@ -129,7 +129,7 @@ export default {
       {
         key: 'development',
         text: '开发者',
-        link: '/zh-cn/docs/development/developers.html',
+        link: '/zh-cn/docs/development/architecture-design.html',
       },
       {
         key: 'community',
diff --git a/src/components/header/index.jsx b/src/components/header/index.jsx
index 14672b1..cd0e993 100755
--- a/src/components/header/index.jsx
+++ b/src/components/header/index.jsx
@@ -48,6 +48,7 @@ class Header extends React.Component {
   constructor(props) {
     super(props);
     this.state = {
+      current: '',
       menuBodyVisible: false,
       language: props.language,
       search: siteConfig.defaultSearch,
@@ -69,15 +70,6 @@ class Header extends React.Component {
       }
     });
   }
-  state = {
-    current: 'home',
-  }
-
-  handleClick = e => {
-    this.setState({
-      current: e.key,
-    });
-  }
 
   componentWillReceiveProps(nextProps) {
     this.setState({
@@ -85,9 +77,11 @@ class Header extends React.Component {
     });
   }
 
-  toggleMenu() {
+  
+
+  handleClick = e => {
     this.setState({
-      menuBodyVisible: !this.state.menuBodyVisible,
+      current: e.key,
     });
   }
 
@@ -206,6 +200,7 @@ class Header extends React.Component {
               onClick={this.toggleMenu}
               src={type === 'primary' ? getLink('/img/system/menu_white.png') : getLink('/img/system/menu_gray.png')}
             />
+            <div>
             <Menu className={type === 'primary'? 'whiteClass': 'blackClass'} onClick={this.handleClick}
selectedKeys={[this.state.current]} mode="horizontal">
             {siteConfig[language].pageMenu.map(item => (
               item.children ? <SubMenu
@@ -217,7 +212,7 @@ class Header extends React.Component {
             >
             <Menu.ItemGroup>
             {item.children.map(items => (
-              <Menu.Item key={items.key}><a href={getLink(items.link)} target={items.target
|| '_self'}>{items.text}</a></Menu.Item>
+              <Menu.Item key={items.key} ><a href={getLink(items.link)} target={items.target
|| '_self'}>{items.text}</a></Menu.Item>
             ))}
             </Menu.ItemGroup>
           </SubMenu> : <Menu.Item key={item.key}>
@@ -226,6 +221,7 @@ class Header extends React.Component {
             ))}
           </Menu>
           </div>
+          </div>
         </div>
       </header>
     );
diff --git a/src/pages/blogDetail/index.scss b/src/pages/blogDetail/index.scss
index b2c9cd1..20dee13 100755
--- a/src/pages/blogDetail/index.scss
+++ b/src/pages/blogDetail/index.scss
@@ -6,6 +6,6 @@
   .blog-content {
     padding: 80px 20%;
     margin: $headerHeight auto 0;
-    max-width: 735px;
+    min-width: 735px;
   }
 }
\ No newline at end of file


Mime
View raw message