54c5e47fc8
It doesn't compile yet, we need to properly link some more encryption libs for it to work. Close though. What sqlite3MultipleCiphers does during compilation is the following: 1. They have a patched sqlite3patched.c amalgamation file, which adds a few lines to the stock amalgamation. I added sqlite3-mc-diff file for reference, that's the difference. 2. They have a sqlite3mc.c file, which is a "reverse amalgamation", a central file that #includes lots of .h and .c files. This file also #includes the sqlite3.c amalgamation. What I did in order to play with sqlite3MultipleCiphers is: 1. Applied the diff above to our sqlite3.c file - applies cleanly, good. 2. Removed "#include sqlite3patched.c" from their sqlite3mc.c file, and instead copy-pasted sqlite3mc.c to the end of our sqlite3.c amalgamation -- not great, but good for starters. 3. It compiles fine, but then fails the linking stage because of some missing encryption function symbols, so it's either because I did something wrong when copying over the C code, or we need to link with external libraries. We can also try it the other way round, so: 1. Apply the diff above to our sqlite3.c file as before 2. Copy over *all* of the sqlite3mc* source files to libsql-ffi/bundled/src 3. Build sqlite3mc.c instead of our sqlite3.c 4. Use it I also managed to succeed in building their shell by running ``` cd build cmake .. -DCMAKE_BUILD_TYPE=Release make -j12 ``` , and it works nicely, and it also properly encrypts WAL files. The `PRAGMA key = something` interface is quite brilliant. Good luck anyone taking over this effort! (And by anyone I mean Lucio, as always)
What is libSQL?
libSQL is an open source, open contribution fork of SQLite, created and maintained by Turso. We aim to evolve it to suit many more use cases than SQLite was originally designed for, and plan to use third-party OSS code wherever it makes sense.
libSQL is licensed under an Open Source License, and we adhere to a clear Code of Conduct
Features
- Embedded replicas that allow you to have replicated database inside your app.
- libSQL server for remote SQLite access, similar to PostgreSQL or MySQL
- Supports Rust, JavaScript, Python, Go, and more.
There are also various improvements and extensions to the core SQLite:
ALTER TABLEextension for modifying column types and constraints- Randomized ROWID
- WebAssembly User Defined Functions
- Pass down SQL string to virtual table implementation
- Virtual write-ahead log interface
The comprehensive description can be found here
Getting Started
The project provides two interfaces: the libSQL API, which supports all the features, and the SQLite C API for compatibility.
To get started with the libSQL API:
- JavaScript
- Rust
- Python (experimental)
- Go (experimental)
- C (experimental)
To build the SQLite-compatible C library and tools, run:
cargo xtask build
To run the SQL shell, launch the libsql program:
$ cd libsql-sqlite3 && ./libsql
libSQL version 0.2.1 (based on SQLite version 3.43.0) 2023-05-23 11:47:56
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
libsql>
Why a fork?
SQLite has solidified its place in modern technology stacks, embedded in nearly any computing device you can think of. Its open source nature and public domain availability make it a popular choice for modification to meet specific use cases.
But despite having its code available, SQLite famously doesn't accept external contributors and doesn't adhere to a code of conduct. So community improvements cannot be widely enjoyed.
There have been other forks in the past, but they all focus on a specific technical difference. We aim to be a community where people can contribute from many different angles and motivations.
We want to see a world where everyone can benefit from all of the great ideas and hard work that the SQLite community contributes back to the codebase. Community contributions work well, because we’ve done it before. If this was possible, what do you think SQLite could become?
You can read more about our goals an motivation in our product vision and our announcement article
Compatibility with SQLite
Compatibility with SQLite is of great importance for us. But it can mean many things. So here's our stance:
- The file format: libSQL will always be able to ingest and write the SQLite file format. We would love to add extensions like encryption, and CRC that require the file to be changed. But we commit to always doing so in a way that generates standard sqlite files if those features are not used.
- The API: libSQL will keep 100% compatibility with the SQLite API, but we may add additional APIs.
- Embedded: SQLite is an embedded database that can be consumed as a single .c file with its accompanying header. libSQL will always be embeddable, meaning it runs inside your process without needing a network connection. But we may change the distribution, so that object files are generated, instead of a single .c file.