oushu/lava-oushudb-dt-sql-parser
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: update docs (#224)

Browse Source 
* docs: change authours field

* docs: upgrade docs
This commit is contained in:
Hayden 2023-11-29 14:56:52 +08:00 committed by GitHub
parent 329f161d2d
commit 7c75164449
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 174 additions and 178 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

155
README-zh_CN.md
View File

@ -1,42 +1,48 @@
# dt-sql-parser # dt-sql-parser
[![NPM version][npm-image]][npm-url] [![NPM version][npm-image]][npm-url] [![NPM downloads][download-img]][download-url] [![Chat][online-chat-img]][online-chat-url]
[npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square [npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square
[npm-url]: https://www.npmjs.com/package/dt-sql-parser [npm-url]: https://www.npmjs.com/package/dt-sql-parser
[download-img]: https://img.shields.io/npm/dm/dt-sql-parser.svg?style=flat
[download-url]: https://www.npmjs.com/package/dt-sql-parser
[online-chat-img]: https://img.shields.io/discord/920616811261743104?logo=Molecule
[online-chat-url]: https://discord.gg/uVvq6mfPfa
[English](./README.md) | 简体中文 [English](./README.md) | 简体中文
dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的， 针对大数据领域的 **SQL Parser** 项目。通过[ANTLR4](https://github.com/antlr/antlr4) 默认生成的 Parser、Visitor 和 Listener 对象，我们可以轻松的做到对 SQL 语句的**语法检查**（Syntax Validation）、**词法分析**（Tokenizer)、 **遍历 AST** 节点等功能。此外，还提供了一些辅助方法, 例如 **SQL 切割（Split）**、**自动补全**等。 dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的， 针对大数据领域的 **SQL Parser** 项目。通过[ANTLR4](https://github.com/antlr/antlr4) 生成的 Parser、Visitor 和 Listener，我们可以轻松的做到对 SQL 语句的**语法检查**（Syntax Validation）、**词法分析**（Tokenizer)、 **遍历 AST** 节点等功能。此外，还提供了一些辅助方法, 例如 **SQL 切割（Split）**、**自动补全**等。
**已支持的 SQL 类型：** **已支持的 SQL 类型：**
- Generic SQL (MySQL) - MySQL
- Flink SQL - Flink SQL
- Spark SQL - Spark SQL
- Hive SQL - Hive SQL
- PL/SQL
- PostgreSQL - PostgreSQL
- Trino SQL - Trino SQL
- Impala SQL
**SQL 辅助方法支持** **SQL 辅助方法支持**
| SQL 类型 | SQL 切割 | 自动补全 | | SQL 类型 | SQL 切割 | 自动补全 |
| ----------- | -------- | -------- | | ----------- | -------- | -------- |
| Generic SQL | WIP | WIP | | MySQL | ✅ | ✅ |
| Flink SQL | ✅ | ✅ | | Flink SQL | ✅ | ✅ |
| Spark SQL | ✅ | ✅ | | Spark SQL | ✅ | ✅ |
| Hive SQL | ✅ | ✅ | | Hive SQL | ✅ | ✅ |
| PL/SQL | WIP | WIP | | PostgreSQL | ✅ | ✅ |
| PostgreSQL | ✅ | ✅ | | Trino SQL | ✅ | ✅ |
| Trino SQL | ✅ | ✅ | | Impala SQL | ✅ | ✅ |
> 提示：当前的 Parser 是 `Javascript` 语言版本，如果有必要，可以尝试编译 Grammar 文件到其他目标语言 > 提示：当前的 Parser 是 `Javascript` 语言版本，如果有必要，可以尝试编译 Grammar 文件到其他目标语言。
<br/> <br/>
## 与 MonacoEditor 集成 ## 与 MonacoEditor 集成
我们提供了一个[monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages)包，你可以轻易的将`dt-sql-parser`与`monaco-editor`集成。 我们提供了[monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages)，通过它你可以轻易的将`dt-sql-parser`与`monaco-editor`集成。
<br/> <br/>
@ -55,28 +61,28 @@ yarn add dt-sql-parser
## 使用 ## 使用
在开始使用前，需要先了解基本用法。`dt-sql-parser` 为不同类型的 SQL 分别提供相应的 SQL Parser 类： 在开始使用前，需要先了解基本用法。`dt-sql-parser` 为不同类型的 SQL 分别提供相应的 SQL Parser 类：
```javascript ```javascript
import { GenericSQL, FlinkSQL, SparkSQL, HiveSQL, PLSQL, PostgresSQL, TrinoSQL } from 'dt-sql-parser'; import { MySQL, FlinkSQL, SparkSQL, HiveSQL, PostgresSQL, TrinoSQL, ImpalaSQL } from 'dt-sql-parser';
``` ```
在使用语法校验，自动补全等功能之前，需要先实例化对应 SQL 类型的 Parser，以 `GenericSQL` 为例： 在使用语法校验，自动补全等功能之前，需要先实例化对应 SQL 类型的 Parser，以 `MySQL` 为例：
```javascript ```javascript
const parser = new GenericSQL(); const parser = new MySQL();
``` ```
下文中的使用示例将使用 `GenericSQL`，其他 SQL 类型的 Parser 使用方式与`GenericSQL` 相同。 下文中的使用示例将使用 `MySQL`，其他 SQL 类型的 Parser 使用方式与`MySQL` 相同。
### 语法校验（Syntax Validation） ### 语法校验（Syntax Validation）
```javascript ```javascript
import { GenericSQL } from 'dt-sql-parser'; import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL(); const parser = new MySQL();
const correctSql = 'select id,name from user1;'; const correctSql = 'select id,name from user1;';
const errors = parser.validate(correctSql); const errors = parser.validate(correctSql);
console.log(errors); console.log(errors);
``` ```
输出： *输出：*
```javascript ```javascript
/* /*
@ -84,7 +90,7 @@ console.log(errors);
*/ */
``` ```
校验失败示例： **校验失败示例：**
```javascript ```javascript
const incorrectSql = 'selec id,name from user1;' const incorrectSql = 'selec id,name from user1;'
@ -92,18 +98,18 @@ const errors = parser.validate(incorrectSql);
console.log(errors); console.log(errors);
``` ```
输出： *输出：*
```javascript ```javascript
/* /*
[ [
{ {
endCol: 5, endCol: 5,
endLine: 1, endLine: 1,
startCol: 0, startCol: 0,
startLine: 1, startLine: 1,
message: "mismatched input 'SELEC' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" message: "mismatched input 'SELEC' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}"
} }
] ]
*/ */
``` ```
@ -112,29 +118,34 @@ console.log(errors);
### 词法分析（Tokenizer） ### 词法分析（Tokenizer）
必要场景下，可单独对 SQL 语句进行词法分析，获取所有的 Tokens 对象： 部分场景下，可以通过 `getAllTokens` 单独对 SQL 语句进行词法分析，获取所有的 Tokens 对象：
```javascript ```javascript
import { GenericSQL } from 'dt-sql-parser'; import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL() const parser = new MySQL()
const sql = 'select id,name,sex from user1;' const sql = 'select id,name,sex from user1;'
const tokens = parser.getAllTokens(sql) const tokens = parser.getAllTokens(sql)
console.log(tokens) console.log(tokens)
```
*输出：*
```javascript
/* /*
[ [
{ {
channel: 0 channel: 0
column: 0 column: 0
line: 1 line: 1
source: [SqlLexer, InputStream] source: [SqlLexer, InputStream]
start: 0 start: 0
stop: 5 stop: 5
tokenIndex: -1 tokenIndex: -1
type: 137 type: 137
_text: null _text: null
}, },
... ...
] ]
*/ */
``` ```
@ -144,13 +155,13 @@ console.log(tokens)
使用 Visitor 模式访问 AST 中的指定节点 使用 Visitor 模式访问 AST 中的指定节点
```javascript ```javascript
import { GenericSQL, SqlParserVisitor } from 'dt-sql-parser'; import { MySQL, MySqlParserVisitor } from 'dt-sql-parser';
const parser = new GenericSQL() const parser = new MySQL()
const sql = `select id,name from user1;` const sql = `select id,name from user1;`
// parseTree // parseTree
const tree = parser.parse(sql) const tree = parser.parse(sql)
class MyVisitor extends SqlParserVisitor { class MyVisitor extends MySqlParserVisitor {
// 重写 visitTableName 方法 // 重写 visitTableName 方法
visitTableName(ctx) { visitTableName(ctx) {
let tableName = ctx.getText().toLowerCase() let tableName = ctx.getText().toLowerCase()
@ -164,12 +175,15 @@ class MyVisitor extends SqlParserVisitor {
} }
const visitor = new MyVisitor() const visitor = new MyVisitor()
visitor.visit(tree) visitor.visit(tree)
```
*输出：*
```javascript
/* /*
SelectElements id,name SelectElements id,name
TableName user1 TableName user1
*/ */
``` ```
> 提示：使用 Visitor 模式时，节点的方法名称可以在对应 SQL 目录下的 Visitor 文件中查找 > 提示：使用 Visitor 模式时，节点的方法名称可以在对应 SQL 目录下的 Visitor 文件中查找
@ -179,13 +193,13 @@ TableName user1
Listener 模式，利用 [ANTLR4](https://github.com/antlr/antlr4) 提供的 `ParseTreeWalker` 对象遍历 AST，进入各个节点时调用对应的方法。 Listener 模式，利用 [ANTLR4](https://github.com/antlr/antlr4) 提供的 `ParseTreeWalker` 对象遍历 AST，进入各个节点时调用对应的方法。
```javascript ```javascript
import { GenericSQL, SqlParserListener } from 'dt-sql-parser'; import { MySQL, MySqlParserListener } from 'dt-sql-parser';
const parser = new GenericSQL(); const parser = new MySQL();
const sql = 'select id,name from user1;' const sql = 'select id,name from user1;'
// parseTree // parseTree
const tree = parser.parse(sql) const tree = parser.parse(sql)
class MyListener extends SqlParserListener { class MyListener extends MySqlParserListener {
enterTableName(ctx) { enterTableName(ctx) {
let tableName = ctx.getText().toLowerCase() let tableName = ctx.getText().toLowerCase()
console.log('TableName', tableName) console.log('TableName', tableName)
@ -197,12 +211,15 @@ class MyListener extends SqlParserListener {
} }
const listenTableName = new MyListener(); const listenTableName = new MyListener();
parser.listen(listenTableName, tree); parser.listen(listenTableName, tree);
```
*输出：*
```javascript
/* /*
SelectElements id,name SelectElements id,name
TableName user1 TableName user1
*/ */
``` ```
> 提示：使用 Listener 模式时，节点的方法名称可以在对应 SQL 目录下的 Listener 文件中查找 > 提示：使用 Listener 模式时，节点的方法名称可以在对应 SQL 目录下的 Listener 文件中查找
@ -215,10 +232,14 @@ const parser = new FlinkSQL();
const sql = 'SHOW TABLES;\nSELECT * FROM tb;'; const sql = 'SHOW TABLES;\nSELECT * FROM tb;';
const sqlSlices = parser.splitSQLByStatement(sql); const sqlSlices = parser.splitSQLByStatement(sql);
console.log(sqlSlices) console.log(sqlSlices)
```
*输出：*
```javascript
/* /*
[ [
{ {
startIndex: 0, startIndex: 0,
endIndex: 11, endIndex: 11,
startLine: 1, startLine: 1,
@ -226,8 +247,8 @@ console.log(sqlSlices)
startColumn: 1, startColumn: 1,
endColumn: 12, endColumn: 12,
text: 'SHOW TABLES;' text: 'SHOW TABLES;'
}, },
{ {
startIndex: 13, startIndex: 13,
endIndex: 29, endIndex: 29,
startLine: 2, startLine: 2,
@ -235,17 +256,16 @@ console.log(sqlSlices)
startColumn: 1, startColumn: 1,
endColumn: 17, endColumn: 17,
text: 'SELECT * FROM tb;' text: 'SELECT * FROM tb;'
} }
] ]
*/ */
``` ```
### 自动补全（Code Completion） ### 自动补全（Code Completion）
在 sql 的指定位置上获取自动补全信息，以 `FlinkSQL` 为例： 在 sql 文本的指定位置上获取自动补全信息，以 `FlinkSQL` 为例：
调用 `getSuggestionAtCaretPosition` 方法，传入 sql 内容和需要自动补全的位置的行列号。 调用 `getSuggestionAtCaretPosition` 方法，传入 sql 内容和指定位置的行列号。
+ 获取关键字候选项列表 + **获取关键字候选项列表**
```javascript ```javascript
import { FlinkSQL } from 'dt-sql-parser'; import { FlinkSQL } from 'dt-sql-parser';
@ -254,12 +274,15 @@ console.log(sqlSlices)
const pos = { lineNumber: 1, column: 16 }; // 最后一个位置 const pos = { lineNumber: 1, column: 16 }; // 最后一个位置
const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords; const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords;
console.log(keywords); console.log(keywords);
```
*输出：*
```javascript
/* /*
[ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ] [ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ]
*/ */
``` ```
+ 获取语法相关自动补全信息 + **获取语法相关自动补全信息**
```javascript ```javascript
const parser = new FlinkSQL(); const parser = new FlinkSQL();
const sql = 'SELECT * FROM tb'; const sql = 'SELECT * FROM tb';
@ -267,6 +290,12 @@ console.log(sqlSlices)
const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax; const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax;
console.log(syntaxSuggestions); console.log(syntaxSuggestions);
```
*输出：*
```javascript
/* /*
[ [
{ {
@ -298,7 +327,7 @@ console.log(sqlSlices)
] ]
*/ */
``` ```
语法相关自动补全信息返回一个数组，数组中每一项代表该位置可以填写什么语法，比如上例中的输出结果代表该位置可以填写**表名**或者**视图名称**。其中 `syntaxContextType` 是可以补全的语法类型，`wordRanges` 则是已经填写的内容。 语法相关自动补全信息返回一个数组，数组中每一项代表该位置可以填写什么语法，比如上例中的输出结果代表该位置可以填写**表名**或者**视图名称**。其中 `syntaxContextType` 是可以补全的语法类型，`wordRanges` 是已经填写的内容。
### 其他 API ### 其他 API

145
README.md
View File

@ -2,8 +2,6 @@
[![NPM version][npm-image]][npm-url] [![NPM downloads][download-img]][download-url] [![Chat][online-chat-img]][online-chat-url] [![NPM version][npm-image]][npm-url] [![NPM downloads][download-img]][download-url] [![Chat][online-chat-img]][online-chat-url]
English | [简体中文](./README-zh_CN.md)
[npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square [npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square
[npm-url]: https://www.npmjs.com/package/dt-sql-parser [npm-url]: https://www.npmjs.com/package/dt-sql-parser
@ -13,31 +11,33 @@ English | [简体中文](./README-zh_CN.md)
[online-chat-img]: https://img.shields.io/discord/920616811261743104?logo=Molecule [online-chat-img]: https://img.shields.io/discord/920616811261743104?logo=Molecule
[online-chat-url]: https://discord.gg/uVvq6mfPfa [online-chat-url]: https://discord.gg/uVvq6mfPfa
English | [简体中文](./README-zh_CN.md)
dt-sql-parser is a **SQL Parser** project built with [ANTLR4](https://github.com/antlr/antlr4), and it's mainly for the **BigData** field. The [ANTLR4](https://github.com/antlr/antlr4) generated the basic Parser, Visitor, and Listener, so it's easy to complete the **syntax validation**, **tokenizer**, **traverse** the AST, and so on features. dt-sql-parser is a **SQL Parser** project built with [ANTLR4](https://github.com/antlr/antlr4), and it's mainly for the **BigData** field. The [ANTLR4](https://github.com/antlr/antlr4) generated the basic Parser, Visitor, and Listener, so it's easy to complete the **syntax validation**, **tokenizer**, **traverse** the AST, and so on features.
Additionally, it provides auxiliary functions such as **SQL splitting** and **code completion**. Additionally, it provides auxiliary functions such as **SQL splitting** and **code completion**.
**Supported SQL**: **Supported SQL**:
- Generic SQL (MySQL) - MySQL
- Flink SQL - Flink SQL
- Spark SQL - Spark SQL
- Hive SQL - Hive SQL
- PL/SQL
- PostgreSQL - PostgreSQL
- Trino SQL - Trino SQL
- Impala SQL
**Supported auxiliary methods** **Supported auxiliary methods**
| SQL Type | SQL Spliting | Code Completion | | SQL Type | SQL Spliting | Code Completion |
| ----------- | ------------ | --------------- | | ----------- | ------------ | --------------- |
| Generic SQL | WIP | WIP | | MySQL | ✅ | ✅ |
| Flink SQL | ✅ | ✅ | | Flink SQL | ✅ | ✅ |
| Spark SQL | ✅ | ✅ | | Spark SQL | ✅ | ✅ |
| Hive SQL | ✅ | ✅ | | Hive SQL | ✅ | ✅ |
| PL/SQL | WIP | WIP | | PostgreSQL | ✅ | ✅ |
| PostgreSQL | ✅ | ✅ | | Trino SQL | ✅ | ✅ |
| Trino SQL | ✅ | ✅ | | Impala SQL | ✅ | ✅ |
>Tips: This project is the default for Javascript language, also you can try to compile it to other languages if you need. >Tips: This project is the default for Javascript language, also you can try to compile it to other languages if you need.
@ -45,8 +45,7 @@ Additionally, it provides auxiliary functions such as **SQL splitting** and **co
## Integrating SQL Parser with Monaco Editor ## Integrating SQL Parser with Monaco Editor
We have provided a [monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages) package, you can integrate with `monaco-editor` We have provided [monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages), it is easily to integrate with `monaco-editor`.
easily.
<br/> <br/>
@ -65,56 +64,57 @@ yarn add dt-sql-parser
## Usage ## Usage
We recommend learning the Fundamentals usage before continuing. The dt-sql-parser library provides SQL Parser classes for different types of SQL. We recommend learning the Fundamentals usage before continuing. The dt-sql-parser library provides SQL Parser classes for different types of SQL.
```javascript ```javascript
import { GenericSQL, FlinkSQL, SparkSQL, HiveSQL, PLSQL, PostgresSQL, TrinoSQL } from 'dt-sql-parser'; import { MySQL, FlinkSQL, SparkSQL, HiveSQL, PostgresSQL, TrinoSQL, ImpalaSQL } from 'dt-sql-parser';
``` ```
Before using syntax validation, code completion, and other features, it is necessary to instantiate the Parser of the relevant SQL type. Before using syntax validation, code completion, and other features, it is necessary to instantiate the Parser of the relevant SQL type.
For instance, one can consider using `GenericSQL` as an example: For instance, one can consider using `MySQL` as an example:
```javascript ```javascript
const parser = new GenericSQL(); const parser = new MySQL();
``` ```
The following usage examples will utilize the `GenericSQL`, and the Parser for other SQL types will be used in a similar manner as `GenericSQL`. The following usage examples will utilize the `MySQL`, and the Parser for other SQL types will be used in a similar manner as `MySQL`.
### Syntax Validation ### Syntax Validation
```javascript ```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic'; import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL(); const parser = new MySQL();
const correctSql = 'select id,name from user1;'; const correctSql = 'select id,name from user1;';
const errors = parser.validate(correctSql); const errors = parser.validate(correctSql);
console.log(errors); console.log(errors);
``` ```
Output: *output:*
```javascript ```javascript
/* /*
[] []
*/ */
``` ```
Validate failed: **Validate failed:**
```javascript ```javascript
const incorrectSql = 'selec id,name from user1;' const incorrectSql = 'selec id,name from user1;'
const errors = parser.validate(incorrectSql); const errors = parser.validate(incorrectSql);
console.log(errors); console.log(errors);
``` ```
Output: *output:*
```javascript ```javascript
/* /*
[ [
{ {
endCol: 5, endCol: 5,
endLine: 1, endLine: 1,
startCol: 0, startCol: 0,
startLine: 1, startLine: 1,
message: "mismatched input 'SELEC' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}" message: "mismatched input 'SELEC' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}"
} }
] ]
*/ */
``` ```
@ -127,26 +127,31 @@ returns an array object includes **error** message.
Get all **tokens** by the Parser: Get all **tokens** by the Parser:
```javascript ```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic'; import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL() const parser = new MySQL()
const sql = 'select id,name,sex from user1;' const sql = 'select id,name,sex from user1;'
const tokens = parser.getAllTokens(sql) const tokens = parser.getAllTokens(sql)
console.log(tokens) console.log(tokens)
```
*output:*
```javascript
/* /*
[ [
{ {
channel: 0 channel: 0
column: 0 column: 0
line: 1 line: 1
source: [SqlLexer, InputStream] source: [SqlLexer, InputStream]
start: 0 start: 0
stop: 5 stop: 5
tokenIndex: -1 tokenIndex: -1
type: 137 type: 137
_text: null _text: null
}, },
... ...
] ]
*/ */
``` ```
@ -156,14 +161,13 @@ console.log(tokens)
Traverse the tree node by the Visitor: Traverse the tree node by the Visitor:
```javascript ```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic'; import { MySQL, MySqlParserVisitor } from 'dt-sql-parser';
import { SqlParserVisitor } from 'dt-sql-parser/dist/parser/generic/SqlParserVisitor';
const parser = new GenericSQL() const parser = new MySQL()
const sql = `select id,name from user1;` const sql = `select id,name from user1;`
// parseTree // parseTree
const tree = parser.parse(sql) const tree = parser.parse(sql)
class MyVisitor extends SqlParserVisitor { class MyVisitor extends MySqlParserVisitor {
// overwrite visitTableName // overwrite visitTableName
visitTableName(ctx) { visitTableName(ctx) {
let tableName = ctx.getText().toLowerCase() let tableName = ctx.getText().toLowerCase()
@ -177,12 +181,15 @@ class MyVisitor extends SqlParserVisitor {
} }
const visitor = new MyVisitor() const visitor = new MyVisitor()
visitor.visit(tree) visitor.visit(tree)
```
*output:*
```javascript
/* /*
SelectElements id,name SelectElements id,name
TableName user1 TableName user1
*/ */
``` ```
> Tips: The node's method name can be found in the Visitor file under the corresponding SQL directory > Tips: The node's method name can be found in the Visitor file under the corresponding SQL directory
@ -192,14 +199,13 @@ TableName user1
Access the specified node in the AST by the Listener Access the specified node in the AST by the Listener
```javascript ```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic'; import { MySQL, MySqlParserListener } from 'dt-sql-parser';
import { SqlParserListener } from 'dt-sql-parser/dist/parser/generic/SqlParserListener';
const parser = new GenericSQL(); const parser = new MySQL();
const sql = 'select id,name from user1;' const sql = 'select id,name from user1;'
// parseTree // parseTree
const tree = parser.parse(sql) const tree = parser.parse(sql)
class MyListener extends SqlParserListener { class MyListener extends MySqlParserListener {
enterTableName(ctx) { enterTableName(ctx) {
let tableName = ctx.getText().toLowerCase() let tableName = ctx.getText().toLowerCase()
console.log('TableName', tableName) console.log('TableName', tableName)
@ -212,11 +218,15 @@ class MyListener extends SqlParserListener {
const listenTableName = new MyListener(); const listenTableName = new MyListener();
parser.listen(listenTableName, tree); parser.listen(listenTableName, tree);
```
*output:*
```javascript
/* /*
SelectElements id,name SelectElements id,name
TableName user1 TableName user1
*/ */
``` ```
> Tips: The node's method name can be found in the Listener file under the corresponding SQL directory > Tips: The node's method name can be found in the Listener file under the corresponding SQL directory
@ -229,10 +239,14 @@ const parser = new FlinkSQL();
const sql = 'SHOW TABLES;\nSELECT * FROM tb;'; const sql = 'SHOW TABLES;\nSELECT * FROM tb;';
const sqlSlices = parser.splitSQLByStatement(sql); const sqlSlices = parser.splitSQLByStatement(sql);
console.log(sqlSlices) console.log(sqlSlices)
```
*output:*
```javascript
/* /*
[ [
{ {
startIndex: 0, startIndex: 0,
endIndex: 11, endIndex: 11,
startLine: 1, startLine: 1,
@ -240,8 +254,8 @@ console.log(sqlSlices)
startColumn: 1, startColumn: 1,
endColumn: 12, endColumn: 12,
text: 'SHOW TABLES;' text: 'SHOW TABLES;'
}, },
{ {
startIndex: 13, startIndex: 13,
endIndex: 29, endIndex: 29,
startLine: 2, startLine: 2,
@ -249,10 +263,9 @@ console.log(sqlSlices)
startColumn: 1, startColumn: 1,
endColumn: 17, endColumn: 17,
text: 'SELECT * FROM tb;' text: 'SELECT * FROM tb;'
} }
] ]
*/ */
``` ```
### Code Completion ### Code Completion
@ -260,7 +273,7 @@ Obtaining code completion information at a specified position in SQL.
We can refer to the example of using `FlinkSQL`. We can refer to the example of using `FlinkSQL`.
Invoke the `getSuggestionAtCaretPosition` method, pass the SQL content and the row and column numbers indicating the position where code completion is desired. Invoke the `getSuggestionAtCaretPosition` method, pass the SQL content and the row and column numbers indicating the position where code completion is desired.
+ keyword candidates list + **keyword candidates list**
```javascript ```javascript
import { FlinkSQL } from 'dt-sql-parser'; import { FlinkSQL } from 'dt-sql-parser';
@ -269,19 +282,23 @@ Invoke the `getSuggestionAtCaretPosition` method, pass the SQL content and the r
const pos = { lineNumber: 1, column: 16 }; // the end position const pos = { lineNumber: 1, column: 16 }; // the end position
const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords; const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords;
console.log(keywords); console.log(keywords);
```
*output:*
```javascript
/* /*
[ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ] [ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ]
*/ */
``` ```
+ Obtaining information related to grammar completion + **Obtaining information related to grammar completion**
```javascript ```javascript
const parser = new FlinkSQL(); const parser = new FlinkSQL();
const sql = 'SELECT * FROM tb'; const sql = 'SELECT * FROM tb';
const pos = { lineNumber: 1, column: 16 }; // after 'tb' const pos = { lineNumber: 1, column: 16 }; // after 'tb'
const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax; const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax;
console.log(syntaxSuggestions); console.log(syntaxSuggestions);
```
*output:*
```javascript
/* /*
[ [
{ {

19
docs/Roadmap.md
View File

@ -1,19 +0,0 @@
# RoadMap
## Supported SQL Language
- Generic SQL
<https://github.com/mysql/mysql-workbench/tree/8.0/library/parsers/grammars>
- [x] PLSQL
- [x] Hive SQL
- [x] Unify parser generate to Antlr4
- [ ] Impala SQL
- [ ] Spark SQL
- [ ] FLink SQL
## TODO
- Libra SQL
- TiDB
MySQL Compatible Syntax

31
docs/Tutorials.md
View File

@ -1,31 +0,0 @@
# Tutorials
## Antlr4 installation
## How to extend new grammar
## How to expose Javascript interface in this project
## Integrate with Monaco Editor
## Release
npm run script
```bash
npm run release -- --release-as minor
```
Or
```bash
npm run release -- --release-as 1.1.0
```
## Reference
- <https://tomassetti.me/writing-a-browser-based-editor-using-monaco-and-antlr/>
- [SQL](https://en.wikipedia.org/wiki/SQL)
- [FlinkSQL](https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/CreateTableLikeTest.java)
- [antlr4 grammar](https://github.com/antlr/grammars-v4/tree/master/sql)
- <https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4>

2
package.json
View File

@ -1,6 +1,7 @@
{ {
"name": "dt-sql-parser", "name": "dt-sql-parser",
"version": "4.0.0-beta.4.7", "version": "4.0.0-beta.4.7",
"authors": "DTStack Corporation",
"description": "SQL Parsers for BigData, built with antlr4", "description": "SQL Parsers for BigData, built with antlr4",
"keywords": [ "keywords": [
"sql", "sql",
@ -29,7 +30,6 @@
"lint-fix": "prettier --write '**/*.ts' --config ./.prettierrc", "lint-fix": "prettier --write '**/*.ts' --config ./.prettierrc",
"cleanComment": "node ./scripts/cleanCommentCli.js" "cleanComment": "node ./scripts/cleanCommentCli.js"
}, },
"author": "dt-insight-front",
"license": "MIT", "license": "MIT",
"devDependencies": { "devDependencies": {
"@commitlint/cli": "^17.7.2", "@commitlint/cli": "^17.7.2",