docs: update docs (#224)

* docs: change authours field

* docs: upgrade docs
This commit is contained in:
Hayden 2023-11-29 14:56:52 +08:00 committed by GitHub
parent 329f161d2d
commit 7c75164449
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 174 additions and 178 deletions

View File

@ -1,42 +1,48 @@
# dt-sql-parser
[![NPM version][npm-image]][npm-url]
[![NPM version][npm-image]][npm-url] [![NPM downloads][download-img]][download-url] [![Chat][online-chat-img]][online-chat-url]
[npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square
[npm-url]: https://www.npmjs.com/package/dt-sql-parser
[download-img]: https://img.shields.io/npm/dm/dt-sql-parser.svg?style=flat
[download-url]: https://www.npmjs.com/package/dt-sql-parser
[online-chat-img]: https://img.shields.io/discord/920616811261743104?logo=Molecule
[online-chat-url]: https://discord.gg/uVvq6mfPfa
[English](./README.md) | 简体中文
dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的, 针对大数据领域的 **SQL Parser** 项目。通过[ANTLR4](https://github.com/antlr/antlr4) 默认生成的 Parser、Visitor 和 Listener 对象,我们可以轻松的做到对 SQL 语句的**语法检查**Syntax Validation、**词法分析**Tokenizer)、 **遍历 AST** 节点等功能。此外,还提供了一些辅助方法, 例如 **SQL 切割Split**、**自动补全**等。
dt-sql-parser 是一个基于 [ANTLR4](https://github.com/antlr/antlr4) 开发的, 针对大数据领域的 **SQL Parser** 项目。通过[ANTLR4](https://github.com/antlr/antlr4) 生成的 Parser、Visitor 和 Listener我们可以轻松的做到对 SQL 语句的**语法检查**Syntax Validation、**词法分析**Tokenizer)、 **遍历 AST** 节点等功能。此外,还提供了一些辅助方法, 例如 **SQL 切割Split**、**自动补全**等。
**已支持的 SQL 类型:**
- Generic SQL (MySQL)
- MySQL
- Flink SQL
- Spark SQL
- Hive SQL
- PL/SQL
- PostgreSQL
- Trino SQL
- Impala SQL
**SQL 辅助方法支持**
| SQL 类型 | SQL 切割 | 自动补全 |
| ----------- | -------- | -------- |
| Generic SQL | WIP | WIP |
| MySQL | ✅ | ✅ |
| Flink SQL | ✅ | ✅ |
| Spark SQL | ✅ | ✅ |
| Hive SQL | ✅ | ✅ |
| PL/SQL | WIP | WIP |
| PostgreSQL | ✅ | ✅ |
| Trino SQL | ✅ | ✅ |
| Impala SQL | ✅ | ✅ |
> 提示:当前的 Parser 是 `Javascript` 语言版本,如果有必要,可以尝试编译 Grammar 文件到其他目标语言
> 提示:当前的 Parser 是 `Javascript` 语言版本,如果有必要,可以尝试编译 Grammar 文件到其他目标语言
<br/>
## 与 MonacoEditor 集成
我们提供了一个[monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages),你可以轻易的将`dt-sql-parser`与`monaco-editor`集成。
我们提供了[monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages)通过它你可以轻易的将`dt-sql-parser`与`monaco-editor`集成。
<br/>
@ -55,28 +61,28 @@ yarn add dt-sql-parser
## 使用
在开始使用前,需要先了解基本用法。`dt-sql-parser` 为不同类型的 SQL 分别提供相应的 SQL Parser 类:
```javascript
import { GenericSQL, FlinkSQL, SparkSQL, HiveSQL, PLSQL, PostgresSQL, TrinoSQL } from 'dt-sql-parser';
import { MySQL, FlinkSQL, SparkSQL, HiveSQL, PostgresSQL, TrinoSQL, ImpalaSQL } from 'dt-sql-parser';
```
在使用语法校验,自动补全等功能之前,需要先实例化对应 SQL 类型的 Parser`GenericSQL` 为例:
在使用语法校验,自动补全等功能之前,需要先实例化对应 SQL 类型的 Parser`MySQL` 为例:
```javascript
const parser = new GenericSQL();
const parser = new MySQL();
```
下文中的使用示例将使用 `GenericSQL`,其他 SQL 类型的 Parser 使用方式与`GenericSQL` 相同。
下文中的使用示例将使用 `MySQL`,其他 SQL 类型的 Parser 使用方式与`MySQL` 相同。
### 语法校验Syntax Validation
```javascript
import { GenericSQL } from 'dt-sql-parser';
import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL();
const parser = new MySQL();
const correctSql = 'select id,name from user1;';
const errors = parser.validate(correctSql);
console.log(errors);
```
输出:
*输出:*
```javascript
/*
@ -84,7 +90,7 @@ console.log(errors);
*/
```
校验失败示例:
**校验失败示例:**
```javascript
const incorrectSql = 'selec id,name from user1;'
@ -92,7 +98,7 @@ const errors = parser.validate(incorrectSql);
console.log(errors);
```
输出:
*输出:*
```javascript
/*
@ -112,15 +118,20 @@ console.log(errors);
### 词法分析Tokenizer
必要场景下,可单独对 SQL 语句进行词法分析,获取所有的 Tokens 对象:
部分场景下,可以通过 `getAllTokens` 单独对 SQL 语句进行词法分析,获取所有的 Tokens 对象:
```javascript
import { GenericSQL } from 'dt-sql-parser';
import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL()
const parser = new MySQL()
const sql = 'select id,name,sex from user1;'
const tokens = parser.getAllTokens(sql)
console.log(tokens)
```
*输出:*
```javascript
/*
[
{
@ -144,13 +155,13 @@ console.log(tokens)
使用 Visitor 模式访问 AST 中的指定节点
```javascript
import { GenericSQL, SqlParserVisitor } from 'dt-sql-parser';
import { MySQL, MySqlParserVisitor } from 'dt-sql-parser';
const parser = new GenericSQL()
const parser = new MySQL()
const sql = `select id,name from user1;`
// parseTree
const tree = parser.parse(sql)
class MyVisitor extends SqlParserVisitor {
class MyVisitor extends MySqlParserVisitor {
// 重写 visitTableName 方法
visitTableName(ctx) {
let tableName = ctx.getText().toLowerCase()
@ -164,12 +175,15 @@ class MyVisitor extends SqlParserVisitor {
}
const visitor = new MyVisitor()
visitor.visit(tree)
```
*输出:*
```javascript
/*
SelectElements id,name
TableName user1
*/
```
> 提示:使用 Visitor 模式时,节点的方法名称可以在对应 SQL 目录下的 Visitor 文件中查找
@ -179,13 +193,13 @@ TableName user1
Listener 模式,利用 [ANTLR4](https://github.com/antlr/antlr4) 提供的 `ParseTreeWalker` 对象遍历 AST进入各个节点时调用对应的方法。
```javascript
import { GenericSQL, SqlParserListener } from 'dt-sql-parser';
import { MySQL, MySqlParserListener } from 'dt-sql-parser';
const parser = new GenericSQL();
const parser = new MySQL();
const sql = 'select id,name from user1;'
// parseTree
const tree = parser.parse(sql)
class MyListener extends SqlParserListener {
class MyListener extends MySqlParserListener {
enterTableName(ctx) {
let tableName = ctx.getText().toLowerCase()
console.log('TableName', tableName)
@ -197,12 +211,15 @@ class MyListener extends SqlParserListener {
}
const listenTableName = new MyListener();
parser.listen(listenTableName, tree);
```
*输出:*
```javascript
/*
SelectElements id,name
TableName user1
*/
```
> 提示:使用 Listener 模式时,节点的方法名称可以在对应 SQL 目录下的 Listener 文件中查找
@ -215,7 +232,11 @@ const parser = new FlinkSQL();
const sql = 'SHOW TABLES;\nSELECT * FROM tb;';
const sqlSlices = parser.splitSQLByStatement(sql);
console.log(sqlSlices)
```
*输出:*
```javascript
/*
[
{
@ -238,14 +259,13 @@ console.log(sqlSlices)
}
]
*/
```
### 自动补全Code Completion
在 sql 的指定位置上获取自动补全信息,以 `FlinkSQL` 为例:
在 sql 文本的指定位置上获取自动补全信息,以 `FlinkSQL` 为例:
调用 `getSuggestionAtCaretPosition` 方法,传入 sql 内容和需要自动补全的位置的行列号。
+ 获取关键字候选项列表
调用 `getSuggestionAtCaretPosition` 方法,传入 sql 内容和指定位置的行列号。
+ **获取关键字候选项列表**
```javascript
import { FlinkSQL } from 'dt-sql-parser';
@ -254,12 +274,15 @@ console.log(sqlSlices)
const pos = { lineNumber: 1, column: 16 }; // 最后一个位置
const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords;
console.log(keywords);
```
*输出:*
```javascript
/*
[ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ]
*/
```
+ 获取语法相关自动补全信息
+ **获取语法相关自动补全信息**
```javascript
const parser = new FlinkSQL();
const sql = 'SELECT * FROM tb';
@ -267,6 +290,12 @@ console.log(sqlSlices)
const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax;
console.log(syntaxSuggestions);
```
*输出:*
```javascript
/*
[
{
@ -298,7 +327,7 @@ console.log(sqlSlices)
]
*/
```
语法相关自动补全信息返回一个数组,数组中每一项代表该位置可以填写什么语法,比如上例中的输出结果代表该位置可以填写**表名**或者**视图名称**。其中 `syntaxContextType` 是可以补全的语法类型,`wordRanges` 是已经填写的内容。
语法相关自动补全信息返回一个数组,数组中每一项代表该位置可以填写什么语法,比如上例中的输出结果代表该位置可以填写**表名**或者**视图名称**。其中 `syntaxContextType` 是可以补全的语法类型,`wordRanges` 是已经填写的内容。
### 其他 API

View File

@ -2,8 +2,6 @@
[![NPM version][npm-image]][npm-url] [![NPM downloads][download-img]][download-url] [![Chat][online-chat-img]][online-chat-url]
English | [简体中文](./README-zh_CN.md)
[npm-image]: https://img.shields.io/npm/v/dt-sql-parser.svg?style=flat-square
[npm-url]: https://www.npmjs.com/package/dt-sql-parser
@ -13,31 +11,33 @@ English | [简体中文](./README-zh_CN.md)
[online-chat-img]: https://img.shields.io/discord/920616811261743104?logo=Molecule
[online-chat-url]: https://discord.gg/uVvq6mfPfa
English | [简体中文](./README-zh_CN.md)
dt-sql-parser is a **SQL Parser** project built with [ANTLR4](https://github.com/antlr/antlr4), and it's mainly for the **BigData** field. The [ANTLR4](https://github.com/antlr/antlr4) generated the basic Parser, Visitor, and Listener, so it's easy to complete the **syntax validation**, **tokenizer**, **traverse** the AST, and so on features.
Additionally, it provides auxiliary functions such as **SQL splitting** and **code completion**.
**Supported SQL**:
- Generic SQL (MySQL)
- MySQL
- Flink SQL
- Spark SQL
- Hive SQL
- PL/SQL
- PostgreSQL
- Trino SQL
- Impala SQL
**Supported auxiliary methods**
| SQL Type | SQL Spliting | Code Completion |
| ----------- | ------------ | --------------- |
| Generic SQL | WIP | WIP |
| MySQL | ✅ | ✅ |
| Flink SQL | ✅ | ✅ |
| Spark SQL | ✅ | ✅ |
| Hive SQL | ✅ | ✅ |
| PL/SQL | WIP | WIP |
| PostgreSQL | ✅ | ✅ |
| Trino SQL | ✅ | ✅ |
| Impala SQL | ✅ | ✅ |
>Tips: This project is the default for Javascript language, also you can try to compile it to other languages if you need.
@ -45,8 +45,7 @@ Additionally, it provides auxiliary functions such as **SQL splitting** and **co
## Integrating SQL Parser with Monaco Editor
We have provided a [monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages) package, you can integrate with `monaco-editor`
easily.
We have provided [monaco-sql-languages](https://github.com/DTStack/monaco-sql-languages), it is easily to integrate with `monaco-editor`.
<br/>
@ -65,29 +64,29 @@ yarn add dt-sql-parser
## Usage
We recommend learning the Fundamentals usage before continuing. The dt-sql-parser library provides SQL Parser classes for different types of SQL.
```javascript
import { GenericSQL, FlinkSQL, SparkSQL, HiveSQL, PLSQL, PostgresSQL, TrinoSQL } from 'dt-sql-parser';
import { MySQL, FlinkSQL, SparkSQL, HiveSQL, PostgresSQL, TrinoSQL, ImpalaSQL } from 'dt-sql-parser';
```
Before using syntax validation, code completion, and other features, it is necessary to instantiate the Parser of the relevant SQL type.
For instance, one can consider using `GenericSQL` as an example:
For instance, one can consider using `MySQL` as an example:
```javascript
const parser = new GenericSQL();
const parser = new MySQL();
```
The following usage examples will utilize the `GenericSQL`, and the Parser for other SQL types will be used in a similar manner as `GenericSQL`.
The following usage examples will utilize the `MySQL`, and the Parser for other SQL types will be used in a similar manner as `MySQL`.
### Syntax Validation
```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic';
import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL();
const parser = new MySQL();
const correctSql = 'select id,name from user1;';
const errors = parser.validate(correctSql);
console.log(errors);
```
Output:
*output:*
```javascript
/*
@ -95,15 +94,16 @@ Output:
*/
```
Validate failed:
**Validate failed:**
```javascript
const incorrectSql = 'selec id,name from user1;'
const errors = parser.validate(incorrectSql);
console.log(errors);
```
Output:
*output:*
```javascript
/*
@ -127,12 +127,17 @@ returns an array object includes **error** message.
Get all **tokens** by the Parser:
```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic';
import { MySQL } from 'dt-sql-parser';
const parser = new GenericSQL()
const parser = new MySQL()
const sql = 'select id,name,sex from user1;'
const tokens = parser.getAllTokens(sql)
console.log(tokens)
```
*output:*
```javascript
/*
[
{
@ -156,14 +161,13 @@ console.log(tokens)
Traverse the tree node by the Visitor:
```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic';
import { SqlParserVisitor } from 'dt-sql-parser/dist/parser/generic/SqlParserVisitor';
import { MySQL, MySqlParserVisitor } from 'dt-sql-parser';
const parser = new GenericSQL()
const parser = new MySQL()
const sql = `select id,name from user1;`
// parseTree
const tree = parser.parse(sql)
class MyVisitor extends SqlParserVisitor {
class MyVisitor extends MySqlParserVisitor {
// overwrite visitTableName
visitTableName(ctx) {
let tableName = ctx.getText().toLowerCase()
@ -177,12 +181,15 @@ class MyVisitor extends SqlParserVisitor {
}
const visitor = new MyVisitor()
visitor.visit(tree)
```
*output:*
```javascript
/*
SelectElements id,name
TableName user1
*/
```
> Tips: The node's method name can be found in the Visitor file under the corresponding SQL directory
@ -192,14 +199,13 @@ TableName user1
Access the specified node in the AST by the Listener
```javascript
import GenericSQL from 'dt-sql-parser/dist/parser/generic';
import { SqlParserListener } from 'dt-sql-parser/dist/parser/generic/SqlParserListener';
import { MySQL, MySqlParserListener } from 'dt-sql-parser';
const parser = new GenericSQL();
const parser = new MySQL();
const sql = 'select id,name from user1;'
// parseTree
const tree = parser.parse(sql)
class MyListener extends SqlParserListener {
class MyListener extends MySqlParserListener {
enterTableName(ctx) {
let tableName = ctx.getText().toLowerCase()
console.log('TableName', tableName)
@ -212,11 +218,15 @@ class MyListener extends SqlParserListener {
const listenTableName = new MyListener();
parser.listen(listenTableName, tree);
```
*output:*
```javascript
/*
SelectElements id,name
TableName user1
*/
```
> Tips: The node's method name can be found in the Listener file under the corresponding SQL directory
@ -229,7 +239,11 @@ const parser = new FlinkSQL();
const sql = 'SHOW TABLES;\nSELECT * FROM tb;';
const sqlSlices = parser.splitSQLByStatement(sql);
console.log(sqlSlices)
```
*output:*
```javascript
/*
[
{
@ -252,7 +266,6 @@ console.log(sqlSlices)
}
]
*/
```
### Code Completion
@ -260,7 +273,7 @@ Obtaining code completion information at a specified position in SQL.
We can refer to the example of using `FlinkSQL`.
Invoke the `getSuggestionAtCaretPosition` method, pass the SQL content and the row and column numbers indicating the position where code completion is desired.
+ keyword candidates list
+ **keyword candidates list**
```javascript
import { FlinkSQL } from 'dt-sql-parser';
@ -269,19 +282,23 @@ Invoke the `getSuggestionAtCaretPosition` method, pass the SQL content and the r
const pos = { lineNumber: 1, column: 16 }; // the end position
const keywords = parser.getSuggestionAtCaretPosition(sql, pos)?.keywords;
console.log(keywords);
```
*output:*
```javascript
/*
[ 'CATALOG', 'FUNCTION', 'TEMPORARY', 'VIEW', 'DATABASE', 'TABLE' ]
*/
```
+ Obtaining information related to grammar completion
+ **Obtaining information related to grammar completion**
```javascript
const parser = new FlinkSQL();
const sql = 'SELECT * FROM tb';
const pos = { lineNumber: 1, column: 16 }; // after 'tb'
const syntaxSuggestions = parser.getSuggestionAtCaretPosition(sql, pos)?.syntax;
console.log(syntaxSuggestions);
```
*output:*
```javascript
/*
[
{

View File

@ -1,19 +0,0 @@
# RoadMap
## Supported SQL Language
- Generic SQL
<https://github.com/mysql/mysql-workbench/tree/8.0/library/parsers/grammars>
- [x] PLSQL
- [x] Hive SQL
- [x] Unify parser generate to Antlr4
- [ ] Impala SQL
- [ ] Spark SQL
- [ ] FLink SQL
## TODO
- Libra SQL
- TiDB
MySQL Compatible Syntax

View File

@ -1,31 +0,0 @@
# Tutorials
## Antlr4 installation
## How to extend new grammar
## How to expose Javascript interface in this project
## Integrate with Monaco Editor
## Release
npm run script
```bash
npm run release -- --release-as minor
```
Or
```bash
npm run release -- --release-as 1.1.0
```
## Reference
- <https://tomassetti.me/writing-a-browser-based-editor-using-monaco-and-antlr/>
- [SQL](https://en.wikipedia.org/wiki/SQL)
- [FlinkSQL](https://github.com/apache/flink/blob/master/flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/CreateTableLikeTest.java)
- [antlr4 grammar](https://github.com/antlr/grammars-v4/tree/master/sql)
- <https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4>

View File

@ -1,6 +1,7 @@
{
"name": "dt-sql-parser",
"version": "4.0.0-beta.4.7",
"authors": "DTStack Corporation",
"description": "SQL Parsers for BigData, built with antlr4",
"keywords": [
"sql",
@ -29,7 +30,6 @@
"lint-fix": "prettier --write '**/*.ts' --config ./.prettierrc",
"cleanComment": "node ./scripts/cleanCommentCli.js"
},
"author": "dt-insight-front",
"license": "MIT",
"devDependencies": {
"@commitlint/cli": "^17.7.2",