-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Feature][Docs] Add engine docs (#15)
- Loading branch information
Showing
11 changed files
with
133 additions
and
27 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 0 additions & 4 deletions
4
...docusaurus-plugin-content-docs/current/04-features/03-engine/01-engine-intro.md
This file was deleted.
Oops, something went wrong.
16 changes: 16 additions & 0 deletions
16
...docusaurus-plugin-content-docs/current/04-features/03-engine/01-local-engine.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
--- | ||
id: 'local-engine' | ||
title: 'Local 引擎' | ||
--- | ||
|
||
## 原理解释 | ||
|
||
`Local` 引擎是通过 `Jdbc` 的方式连接数据源,执行根据数据质量检查规则生成的 `SQL` 语句,得到相应的错误数据视图、实际值和期望值,最后将实际值和期望值写到 `Datavines` 系统的执行结果表以及将错误数据视图中的数据写到错误数据存储引擎。 | ||
|
||
## 使用方法 | ||
|
||
`Local` 引擎的使用非常简单,创建数据质量检查作业时默认使用的是 `Local` 引擎,无需做任何额外的配置,保存好配置执行即可,需要关注一下注意事项。 | ||
|
||
## 注意事项 | ||
|
||
- 用于创建数据源的用户需要有创建视图和删除视图的权限,如果你想要将错误数据写入到检查的数据源中某个库中,那么你的用户需要有创建表的权限。 |
4 changes: 0 additions & 4 deletions
4
...docusaurus-plugin-content-docs/current/04-features/03-engine/02-local-engine.md
This file was deleted.
Oops, something went wrong.
23 changes: 23 additions & 0 deletions
23
...docusaurus-plugin-content-docs/current/04-features/03-engine/02-spark-engine.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
--- | ||
id: 'spark-engine' | ||
title: 'Spark 引擎' | ||
--- | ||
|
||
## 原理解释 | ||
|
||
`Spark` 引擎是基于 `Spark` 开发的数据质量检查作业执行引擎,SparkDataVinesBootstrap 是应用的执行入。`Datavines` 会根据用户所配置各种信息构造成配置文件并传入到 `SparkDataVinesBootstrap`中,`SparkDataVinesBootstrap` 会解析配置文件并选择不同的 `source`、`transform` 和 `sink` 插件来执行连接数据源、质量检查语句和将执行结果写入到相应的存储引擎中的操作。 | ||
|
||
## 使用方法 | ||
|
||
在数据质量检查作业和数据比对作业中的引擎配置中选择 `Spark` 引擎,填好各种 `Spark` 相关的基础配置信息 | ||
![数据质量检查规则引擎配置](/doc/image/metric_job_engine.png) | ||
|
||
## 注意事项 | ||
- 如果想要使用 `Spark` 引擎,那么需要把 `Datavines` 部署在能执行 `spark-submit` 的服务器上 | ||
- 需要配置可以执行 `spark-submit` 的服务器用户 和 环境配置 | ||
- 服务器用户指的是 `root`、`hadoop`、`ods`等等这些服务器创建的用户 | ||
- 环境配置指的是 在服务器上 export 的各种配置 | ||
``` | ||
export SPARK_HOME2=/usr/hdp/2.6.3.0-235/spark-2.4.8-bin-hadoop2.7 | ||
``` | ||
![数据质量检查规则环境配置](/doc/image/metric_job_env.png) |
4 changes: 0 additions & 4 deletions
4
...docusaurus-plugin-content-docs/current/04-features/03-engine/03-spark-engine.md
This file was deleted.
Oops, something went wrong.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.