* add basic functions for working with f32/f64 vectors
* add header file
* fix comment style
* add generic code for working with vectors
* register vector functions
* fix edge cases in vector extract impl
* add static flag for vector (will be used later)
* add basic TCL-based tests
* guard vector functions with SQLITE_OMIT_VECTOR
* adjust build scripts
* delete libsql_vector_idx from this branch
* add one more test
* update bundles
* fix asserts
* fix constants (#define-s are universal but const int doesn't recognized by all compilers)
* update bundles
* review fixes
* build bundles
* add comment about linking math library
* add alter column test
* apply alter column only to the table and ignore indices/triggers/views etc
* fix tests
* update bundle
* add test in rust_suite
* add basic TCL tests for ALTER COLUMN libsql feature
* propagate length of the new column definition directly from the parser
- this will allow libsql to automatically handle comment and space characters appended to the column definition
* small formatting fixes
* cargo xtask build-bundled
* add test against sqlite3 source in rust_suite
* fix test in rust_suite
* make sure pager is valid
We have had a segfault issue recently because the pager passed down
to the c code was null. That happened because libsql-server had
encryption disabled, while libsql had encryption enabled.
To prevent issues like that in the future, the C code should be checking
if the arguments it needs are valid.
Because this function returned a pointer, we now change it so that it
can return an integer, with the pointer as a return argument.
* update generated code
We have generated code in-tree. We should either remove it, or keep it
in sync. For now, just keeping it in sync.
This patch is just the result of running tests, and seeing what changes.
* test snapshots
* allow multi-scope auth
* hors-sujet: fix C compile warning
* update jwt and add hashbrown feature
* replace Auth with RequestContext
* extract RequestContext
* fix auth check for multi-scopes
* pass NamespaceName in StmtKind::Attach
* fallback to global auth if key is not set in ns config
* add allow_attach to namespace creation
* return meta store from namespace store
* test attach
* fmt
* review edits
* Add regression tests for random rowid bugs
Tests added for:
1. VACUUM does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/929
2. Bulk insert into a fresh table does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/1046
In both cases, the table should end up random rowids, but they seem to have sequential ones.
* bugfix: pass `regNextRowid` to `OP_NewRowid` call in `xferOptimization`
The register `regNextRowid` contains the `LIBSQL_RANDOM_ROWID_MARKER` value
which tells the VDBE to use a random value for row ids. The method `xferOptimization`
is used in `VACUUM` and bulk insert routines where data from one table is
inserted in another. Since the data is inserted in a loop, it checks for
max rowid once and then keeps inserting it serially. Hence we pass the marker
in each call, so that random id is generated.
* libsql: Make encryption cipher configurable
Introduce a `EncryptionConfig` struct to configure both encrytion cipher
and key. Needed to support multiple ciphers.
Fixes#951
* libsql-ffi: Switch to SQLCipher as the default cipher
Fixes#893
The commit d178de8f07 fixed ABI breakage, but we need to regenerate the
bundled sqlite3mc code to make sure the bug fix is included when
encryption is enabled.
* Remove `libsql_module` field from `sqlite3_vtab`
* remove `libsql_module` struct
Also removed the associated functions `libsql_create_module_v2`, `libsql_create_module` functions'.
The `libsql_module` had a function `xPreparedSql` which is now moved to `sqlite_module`. The `sqlite_module` might get changed in the upstream, so added some padding space for our custom functions
* generate ffi bindings
* Add a github workflow to test crsqlite
* fix crsqlite: remove `pLibsqlModule` references
* Add tests for sqlite-vss extension
* libsql-ffi: Enable PIC for sqlite3mc build
I am seeing the following build error when building the libsql-ffi crate
on Linux/x86-64:
```
/usr/bin/ld:
/tmp/rustcYVhHYo/liblibsql_ffi-20adf4ca11d8e922.rlib(sqlite3mc.c.o):
relocation R_X86_64_32S against `.rodata' can not be used when making a
shared object; recompile with -fPIC
```
Let's turn on PIC for sqlite3mc to fix the problem.
* libsql-ffi: Fix sqlite3mc build under `cross`
We need to use the C compiler configured by `cross` with sqlite3mc;
otherwise we end up building with the host compiler, resulting in errors
like this:
```
= note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcJ70WRo/liblibsql_ffi-5d7e6e8037dbda37.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
```
* libsql-ffi: Rewrite build_libsqlite3mc.sh in Rust
The bash shell script is turning out to be painful to run on Windows so
let's just rewrite the logic in Rust and place it in `build.rs`.
* namespace,replication: add LogFile encryption
Anything that uses our LogFile format can now be encrypted
on-disk.
Tested locally by seeing that `wallog` file contains garbage
and no sensible plaintext strings can be extracted from it.
* test fixups
* libsql-ffi: add libsql_generate_initial_vector and...
... libsql_generate_aes256_key to make them reachable from Rust.
* connection: expose additional encryption symbols
* libsql-server: derive aes256 from user passphrase properly
And by properly, I mean calling back to SQLite3MultipleCiphers' code.
* replication: rename Encryptor to FrameEncryptor
Encryptor sounds a little too generic for this specific use case.
* replication: add snapshot encryption
It uses the same mechanism as wallog encryption, now abstracted
away to libsql-replication crate to be reused.
* replication: add an encryption feature for compilation
* cargo fmt pass
* fix remaining SnapshotFile::open calls in tests
* logger: add an encryption test
* replication: use a single buffer for encryption
Ideally we could even encrypt in place, but WalPage is also
used in snapshots and it's buffered, and that makes it exceptionally
annoying to explain to the borrow checker.
* bottomless: restore with libsql_replication::injector
... instead of the transaction page cache. That gives us free
encryption, since the injector is encryption-aware.
This patch doesn't hook encryption_key parameter yet, it will
come in the next patch.
* bottomless: pass the encryption key in options
For WAL restoration, but also to be able to encrypt data that gets
sent to S3.
* bottomless: inherit encryption key from db config if not specified
* libsql-sys: add db_change_counter()
The helper function calls the underlying C API to extract
4 bytes from offset 24 of the database header and return it.
It's the database change counter, which we can use to compare
two databases and decide which one is newer than the other.
* bottomless: use sqlite API to read database metadata
With encryption enabled, we can no longer just go ahead and read data
from given offsets, we must go through the VFS layer instead.
Fortunately, we can just open a database connection and ask for all
the metadata we need.
* libsql-sys: make db change counter actually read from the db file
* bottomless: treat change counter == 1 as a new database
... which it is, after setting the journal mode. Otherwise we decide
too eagerly that the local database is the source of truth.
* libsql-server: fix a local embedded replica test
rebase conflict with encryption
* bottomless-cli: allow passing the encryption key
* replication: rebase new test to the new api
* snapshots: do not try to decrypt headers
They are not encrypted, so we shouldn't attempt to decrypt the data.
* logger: restore encrypted frames during recovery
Instead of decrypting and encrypting back, we just copy encrypted
frames as is during the recovery process, saves IO.
* compaction: clear unused encryption_key parameter
It wasn't used since for compaction we only need headers,
which are unencrypted.
* replication: switch to FrameBorrowed::new_zeroed
Following MarinPostma's suggestion.
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
* replication: rebase chores, fixing parameters
* libsql-replication: use page_mut() to decrypt data in-place
* rustfmt
* bottomless: use 0 for disabling autocheckpoint
... instead of u32::MAX. Effectively it's similar, but 0 is the correct
choice.
* rustfmt
* libsql-server: make cbc, aes optional for encryption only
* post-rebase fixes
* libsql-replication: suppress warnings when no encryption
* libsql: add encryption support for local databases
* libsql: add bytes dependency for encryption
* libsql-ffi: build libsqlite3mc without debug symbols
Technically it should just depend on cargo build mode,
but that's left for a follow-up.
* bindings: an attempt to compile bindings with releasemode
... partially to save space, but also to make them faster.
---------
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
The CMake definitions look like they only apply to builds which enable
shell, which doesn't make much sense. We rely on dbstat, so let's just
enable it.
* wal.c: fix the function pointer type
A 100% benign error, because the function has the correct signature,
but gets rid of a compiler warning.
* regenerate sqlite3.c amalgamation
* libsql-ffi: regenerate SQLite3MultipleCiphers' sqlite3.c
... not needed later, because an existing patch series switches it
to use the sqlite3.c amalgamation file from bundled/src/,
but let's update it for now.
- add the checkpoint callback to Wal::checkpoint
- use dynamic dispatch for callbacks (correctness issue)
- pass `frames_in_wal` and `backfilled` as ref to `Wal::checkpoint`
because sqlite can set them despite returning `SQLITE_BUSY`
** DEEP, DEEP DRAFT **
The prebuilt library is from my fork:
https://github.com/psarna/SQLite3MultipleCiphers
The key is hardcoded to "heyhey".
After you run sqld with this patch, all data is encoded on disk
with a "heyhey" key. You can't read it directly from the file,
unless you use sqlite3mc's shell and start with
> PRAGMA KEY=heyhey;
, and then it gets properly decrypted.
** TODO **
1. We need to adopt SQLite3MultipleCiphers source code and integrate
with our build system, if we want to use it.
2. Pretty sure the hardcoded "heyhey" passphrase won't pass SOC2,
but I need to consult that with a lawyer.
