From fa5fed2c2aa8eddf1080078d48efbcac38cf396c Mon Sep 17 00:00:00 2001 From: xiaowei Date: Thu, 17 Dec 2020 16:46:42 +0800 Subject: [PATCH] docs: update README --- NeREADME.md | 33 ---------- README-zh_CN.md | 162 +++++++++++++++++++++++++----------------------- README.md | 157 +++++++++++++++++++++++++--------------------- 3 files changed, 172 insertions(+), 180 deletions(-) delete mode 100644 NeREADME.md diff --git a/NeREADME.md b/NeREADME.md deleted file mode 100644 index 0492d02..0000000 --- a/NeREADME.md +++ /dev/null @@ -1,33 +0,0 @@ -# dt-sql-parser - -[![NPM version][npm-image]][npm-url] - -[npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square -[npm-url]: https://www.npmjs.com/package/dt-sql-parser - -## Installation - -## Usage - -### Basic - -### Syntax validation - -### Visitor - -### Listener - -## Example - -## Roadmap - -- Unify parser generate to Antlr4 -- Generic SQL -- Flink SQL -- Libra SQL -- TiDB - MySQL Compatible Syntax - -## Contributing - -## License diff --git a/README-zh_CN.md b/README-zh_CN.md index 0d47677..c883873 100644 --- a/README-zh_CN.md +++ b/README-zh_CN.md @@ -7,15 +7,9 @@ [English](./README.md) | 简体中文 -dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的 SQL 解析器集合。主要用于大数据开发中，对各类 SQL 的解析。 +dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的， 针对大数据领域的 `SQL Parser` 项目。通过[ANTLR4](https://github.com/antlr/antlr4) 默认生成的 Parser、Visitor 和 Listener 对象，我们可以轻松的做到对 SQL 语句的`语法检查`（Syntax Validation）、`词法分析`（Tokenizer)、 `遍历 AST` 节点等功能。此外，还提供了几个辅助方法, 例如 SQL 切分（Split）、过滤 SQL 语句中的 `--` 和 `/**/` 等类型的注释。 -每种 SQL 都提供了对应基础类、Visitor 类和 Listener 类，包含了生成 token、生成 AST、语法校验、visitor 和 listener 模式遍历 AST 指定节点等功能。 - -此外，为了方便解析，还提供了几个辅助方法可以在解析前对 SQL 进行格式处理。主要作用是清除 SQL 语句中的 '--' 和 '/**/' 两种类型的注释，以及拆分大段 SQL。 - -提示：项目中的 grammar 文件也可以通过 [ANTLR4](https://github.com/antlr/antlr4) 编译成其他语言 - -目前支持的 SQL： +已支持的 SQL 类型： - MySQL - Flink SQL @@ -23,9 +17,11 @@ dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发 - Hive SQL - PL/SQL +> 提示：当前的 Parser 是 `Javascript` 语言版本，如果有必要，可以尝试编译 Grammar 文件到其他目标语言 + ## 安装 -``` +```bash // use npm npm i dt-sql-parser --save @@ -33,45 +29,60 @@ npm i dt-sql-parser --save yarn add dt-sql-parser ``` -## 示例 +## 使用 -### Clean +### 语法校验（Syntax Validation） -清除注释和前后空格 +首先需要声明想对应的 Parser 对象，不同的 SQL 类型需要引入不同的 Parser 对象处理，例如如果是 +真的 `Flink SQL`, 则需要单独引入 `FlinkSQL` 对象, 这里我们使用 `GenericSQL` 作为示例： ```javascript -import { cleanSql } from 'dt-sql-parser'; +import { GenericSQL } from 'dt-sql-parser'; -const sql = `-- comment comment -select id,name from user1; ` -const cleanedSql = cleanSql(sql) -console.log(cleanedSql) +const parser = new GenericSQL(); +const correctSql = 'select id,name from user1;'; +const errors = parser.validate(correctSql); +console.log(errors); +``` + +输出： + +```javascript /* -select id,name from user1; +[] */ ``` -### Split - -分割 sql +校验失败示例： ```javascript -import { splitSql } from 'dt-sql-parser'; +const incorrectSql = 'selec id,name from user1;' +const errors = parser.validate(incorrectSql); +console.log(errors); +``` -const sql = `select id,name from user1; -select id,name from user2;` -const sqlList = splitSql(sql) -console.log(sqlList) +输出： +```javascript /* -["select id,name from user1;", "\nselect id,name from user2;"] +[ + { + endCol: 5, + endLine: 1, + startCol: 0, + startLine: 1, + message: "mismatched input 'SELEC' expecting {, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" + } +] */ ``` -### Tokens +先实例化 Parser 对象，然后使用 `validate` 方法对 SQL 语句进行校验，如果校验失败，则返回一个包含 `Error` 信息的数组。 -对 sql 语句进行词法分析，生成 token +### 词法分析（Tokenizer） + +必要场景下，可单独对 SQL 语句进行词法分析，获取所有的 Tokens 对象： ```javascript import { GenericSQL } from 'dt-sql-parser'; @@ -99,47 +110,9 @@ console.log(tokens) */ ``` -### Syntax validation +### 访问者模式（Visitor） -validate 方法对 sql 语句的语法正确性进行校验，返回一个由 error 组成的数组 - -```javascript -import { GenericSQL } from 'dt-sql-parser'; - -const validate = (sql) => { - const parser = new GenericSQL() - const errors = parser.validate(sql) - console.log(errors) -} -``` -语法正确的 sql: -```javascript -const correctSql = 'select id,name from user1;' -validate(correctSql) -/* -[] -*/ -``` -包含错误语法的 sql: -```javascript -const incorrectSql = 'selec id,name from user1;' -validate(incorrectSql) -/* -[ - { - endCol: 5, - endLine: 1, - startCol: 0, - startLine: 1, - message: "mismatched input 'SELEC' expecting {, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" - } -] -*/ -``` - -### Visitor - -使用 visitor 模式访问 AST 中的指定节点 +使用 Visitor 模式访问 AST 中的指定节点 ```javascript import { GenericSQL, SqlParserVisitor } from 'dt-sql-parser'; @@ -169,11 +142,12 @@ TableName user1 */ ``` -提示：使用 Visitor 模式时，节点的方法名称可以在对应 SQL 目录下的 Visitor 文件中查找 -### Listener +> 提示：使用 Visitor 模式时，节点的方法名称可以在对应 SQL 目录下的 Visitor 文件中查找 -listener 模式，利用 [ANTLR4](https://github.com/antlr/antlr4) 提供的 ParseTreeWalker 对象遍历 AST，进入各个节点时调用对应的方法。 +### 监听器（Listener） + +Listener 模式，利用 [ANTLR4](https://github.com/antlr/antlr4) 提供的 ParseTreeWalker 对象遍历 AST，进入各个节点时调用对应的方法。 ```javascript import { GenericSQL, SqlParserListener } from 'dt-sql-parser'; @@ -202,11 +176,47 @@ TableName user1 ``` -提示：使用 Listener 模式时，节点的方法名称可以在对应 SQL 目录下的 Listener 文件中查找 +> 提示：使用 Listener 模式时，节点的方法名称可以在对应 SQL 目录下的 Listener 文件中查找 -### 其他 +### 清理注释内容 -- parserTreeToString (将 SQL 解析成 AST，再转成 string 形式) +清除注释和前后空格 + +```javascript +import { cleanSql } from 'dt-sql-parser'; + +const sql = `-- comment comment +select id,name from user1; ` +const cleanedSql = cleanSql(sql) +console.log(cleanedSql) + +/* +select id,name from user1; +*/ +``` + +### 切割 SQL （Split） + +SQL 太大的情况下，我们可以先将SQL语句按 `;` 切割，然后逐句处理。 + +```javascript +import { splitSql } from 'dt-sql-parser'; + +const sql = `select id,name from user1; +select id,name from user2;` +const sqlList = splitSql(sql) +console.log(sqlList) + +/* +["select id,name from user1;", "\nselect id,name from user2;"] +*/ +``` + +### 其他 API + +- parserTreeToString (input: string) + +将 SQL 解析成 `List-like` 风格的树形字符串， 一般用于测试 ## 路线图 @@ -215,4 +225,4 @@ TableName user1 ## 许可证 -[MIT](./LICENSE) \ No newline at end of file +[MIT](./LICENSE) diff --git a/README.md b/README.md index 4bc3339..b801aa1 100644 --- a/README.md +++ b/README.md @@ -2,18 +2,16 @@ [![NPM version][npm-image]][npm-url] +English | [简体中文](./README-zh_CN.md) + [npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square [npm-url]: https://www.npmjs.com/package/dt-sql-parser -English | [简体中文](./README-zh_CN.md) +dt-sql-parser is a `SQL Parser` project built with [ANTLR4](https://github.com/antlr/antlr4), and it's mainly for the `BigData` domain. The [ANTLR4](https://github.com/antlr/antlr4) generated the basic Parser, Visitor, and Listener, so it's easy to complete `validate`, `tokenize`, `traverse` the AST, and so on features. -dt-sql-parser is a collection of SQL parsers developed based on [ANTLR4](https://github.com/antlr/antlr4) .It's mainly used for parsing all kinds of SQL in the development of big data. +Besides, it' provides some helper methods, like `split` SQL, and filter the `--` and `/**/` types of comments in SQL. -It provides the basic class, Visitor class, and Listener class. These class including the ability to generate tokens, generate parse tree, syntax validation, and Visitor & Listener patterns to traverse the AST. - -In addition, several helper methods are provided to format the SQL before parsing. The main effect is to clear the '--' and '/**/' types of comments in SQL statements, and to split large chunks of SQL - -tips: The Grammar file can also be compiled into other languages with [ANTLR4](https://github.com/antlr/antlr4) . +> Tips: This project is the default for Javascript language, also you can try to compile it to other languages if you need. Supported SQL: @@ -25,7 +23,7 @@ Supported SQL: ## Installation -``` +```bash // use npm npm i dt-sql-parser --save @@ -35,43 +33,61 @@ yarn add dt-sql-parser ## Usage -### Clean +### Syntax Validation -clear comments and Spaces before and after +First, we need to import the `Parser` object from `dt-sql-parser`, the different language needs +different Parser, so if you need to handle the `Flink SQL`, you can import the `FlinkSQL Parser`. + +The below is a `GenericSQL Parser` example: ```javascript -import { cleanSql } from 'dt-sql-parser'; +import { GenericSQL } from 'dt-sql-parser'; -const sql = `-- comment comment -select id,name from user1; ` -const cleanedSql = cleanSql(sql) -console.log(cleanedSql) +const parser = new GenericSQL(); +const correctSql = 'select id,name from user1;'; +const errors = parser.validate(correctSql); +console.log(errors); +``` + +output: + +```javascript /* -select id,name from user1; +[] */ ``` -### Split - -split sql +validate failed: ```javascript -import { splitSql } from 'dt-sql-parser'; +const incorrectSql = 'selec id,name from user1;' +const errors = parser.validate(incorrectSql); +console.log(errors); +``` -const sql = `select id,name from user1; -select id,name from user2;` -const sqlList = splitSql(sql) -console.log(sqlList) +output: +```javascript /* -["select id,name from user1;", "\nselect id,name from user2;"] +[ + { + endCol: 5, + endLine: 1, + startCol: 0, + startLine: 1, + message: "mismatched input 'SELEC' expecting {, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" + } +] */ ``` -### Tokens +We instanced a Parser object, and use the `validate` method to check the SQL syntax, if failed +returns an array object includes `error` message. -lexical analysis, generate token +### Tokenizer + +You can also the all `tokens` by the Parser: ```javascript import { GenericSQL } from 'dt-sql-parser'; @@ -99,47 +115,9 @@ console.log(tokens) */ ``` -### Syntax validation - -verifies the syntax correctness of the SQL statement and returns an array of errors - -```javascript -import { GenericSQL } from 'dt-sql-parser'; - -const validate = (sql) => { - const parser = new GenericSQL() - const errors = parser.validate(sql) - console.log(errors) -} -``` -correct sql: -```javascript -const correctSql = 'select id,name from user1;' -validate(correctSql) -/* -[] -*/ -``` -incorrect sql: -```javascript -const incorrectSql = 'selec id,name from user1;' -validate(incorrectSql) -/* -[ - { - endCol: 5, - endLine: 1, - startCol: 0, - startLine: 1, - message: "mismatched input 'SELEC' expecting {, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" - } -] -*/ -``` - ### Visitor -access the specified node in the AST by Visitor pattern +Traverse the tree node by the Visitor: ```javascript import { GenericSQL, SqlParserVisitor } from 'dt-sql-parser'; @@ -169,7 +147,8 @@ TableName user1 */ ``` -tips: The node's method name can be found in the Visitor file under the corresponding SQL directory + +> Tips: The node's method name can be found in the Visitor file under the corresponding SQL directory ### Listener @@ -202,11 +181,47 @@ TableName user1 ``` -tips: The node's method name can be found in the Listener file under the corresponding SQL directory +> Tips: The node's method name can be found in the Listener file under the corresponding SQL directory -### Other +### Clean -- parserTreeToString (parse the SQL into AST and turn it into a String) +Clear the `comments` and `spaces` before and after + +```javascript +import { cleanSql } from 'dt-sql-parser'; + +const sql = `-- comment comment +select id,name from user1; ` +const cleanedSql = cleanSql(sql) +console.log(cleanedSql) + +/* +select id,name from user1; +*/ +``` + +### Split SQL + +When the SQL text is very big, you can think about to split it by `;` , and handle each line. + +```javascript +import { splitSql } from 'dt-sql-parser'; + +const sql = `select id,name from user1; +select id,name from user2;` +const sqlList = splitSql(sql) +console.log(sqlList) + +/* +["select id,name from user1;", "\nselect id,name from user2;"] +*/ +``` + +### Other API + +- parserTreeToString(input: string) + +Parse the input and convert the AST to a `List-like` tree string. ## Roadmap @@ -215,4 +230,4 @@ tips: The node's method name can be found in the Listener file under the corresp ## License -[MIT](./LICENSE) \ No newline at end of file +[MIT](./LICENSE)