mirror of
https://github.com/tursodatabase/libsql.git
synced 2024-12-15 21:29:01 +00:00
75 lines
3.0 KiB
Markdown
75 lines
3.0 KiB
Markdown
[](https://github.com/gwenn/lemon-rs/actions)
|
|
[](https://crates.io/crates/sqlite3-parser)
|
|
[](https://docs.rs/sqlite3-parser)
|
|
[](https://deps.rs/repo/github/gwenn/lemon-rs)
|
|
|
|
[LEMON parser generator](https://www.sqlite.org/src/doc/trunk/doc/lemon.html) modified to generate Rust code.
|
|
|
|
Lemon source and SQLite3 grammar were last synced as of May 2022.
|
|
|
|
## Unsupported
|
|
|
|
### Unsupported Grammar syntax
|
|
|
|
* `%token_destructor`: Code to execute to destroy token data
|
|
* `%default_destructor`: Code for the default non-terminal destructor
|
|
* `%destructor`: Code which executes whenever this symbol is
|
|
popped from the stack during error processing
|
|
|
|
https://www.codeproject.com/Articles/1056460/Generating-a-High-Speed-Parser-Part-Lemon
|
|
https://www.sqlite.org/lemon.html
|
|
|
|
### SQLite
|
|
|
|
[SQLite lexer](http://www.sqlite.org/src/artifact?ci=trunk&filename=src/tokenize.c) and [SQLite parser](http://www.sqlite.org/src/artifact?ci=trunk&filename=src/parse.y) have been ported from C to Rust.
|
|
The parser generates an AST.
|
|
|
|
Lexer/Parser:
|
|
- Keep track of position (line, column).
|
|
- Streamable (stop at the end of statement).
|
|
- Resumable (restart after the end of statement).
|
|
|
|
Lexer and parser have been tested with the following scripts:
|
|
* https://github.com/bkiers/sqlite-parser/tree/master/src/test/resources
|
|
* https://github.com/codeschool/sqlite-parser/tree/master/test/sql/official-suite which can be updated with script in https://github.com/codeschool/sqlite-parser/tree/master/test/misc
|
|
|
|
TODO:
|
|
- [ ] Check generated AST (reparse/reinject)
|
|
- [ ] [If a keyword in double quotes is used in a context where it cannot be resolved to an identifier but where a string literal is allowed, then the token is understood to be a string literal instead of an identifier.](https://sqlite.org/lang_keywords.html)
|
|
- [ ] Tests
|
|
- [ ] Do not panic while parsing
|
|
- [ ] CREATE VIRTUAL TABLE args
|
|
- [ ] Zero copy (at least tokens)
|
|
|
|
### Unsupported by Rust
|
|
|
|
* `#line` directive
|
|
|
|
## API change
|
|
|
|
* No `ParseAlloc`/`ParseFree` anymore
|
|
|
|
## Features not tested
|
|
|
|
* NDEBUG
|
|
* YYNOERRORRECOVERY
|
|
* YYERRORSYMBOL
|
|
|
|
## To be fixed
|
|
|
|
* RHS are moved. Maybe it is not a problem if they are always used once.
|
|
Just add a check in lemon...
|
|
* `%extra_argument` is not supported.
|
|
* Terminal symbols generated by lemon should be dumped in a specified file.
|
|
|
|
## Raison d'être
|
|
|
|
* [lemon_rust](https://github.com/rodrigorc/lemon_rust) does the same thing
|
|
but with an old version of `lemon`. And it seems not possible to use `yystack`
|
|
as a stack because items may be access randomly and the `top+1` item can be used.
|
|
|
|
* [lalrpop](https://github.com/nikomatsakis/lalrpop) would be the perfect
|
|
alternative but it does not support fallback/streaming
|
|
(see [this](https://github.com/nikomatsakis/lalrpop/issues/156) issue)
|
|
and compilation/generation is slow.
|