* Add regression tests for random rowid bugs
Tests added for:
1. VACUUM does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/929
2. Bulk insert into a fresh table does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/1046
In both cases, the table should end up random rowids, but they seem to have sequential ones.
* bugfix: pass `regNextRowid` to `OP_NewRowid` call in `xferOptimization`
The register `regNextRowid` contains the `LIBSQL_RANDOM_ROWID_MARKER` value
which tells the VDBE to use a random value for row ids. The method `xferOptimization`
is used in `VACUUM` and bulk insert routines where data from one table is
inserted in another. Since the data is inserted in a loop, it checks for
max rowid once and then keeps inserting it serially. Hence we pass the marker
in each call, so that random id is generated.
* Remove `libsql_module` field from `sqlite3_vtab`
* remove `libsql_module` struct
Also removed the associated functions `libsql_create_module_v2`, `libsql_create_module` functions'.
The `libsql_module` had a function `xPreparedSql` which is now moved to `sqlite_module`. The `sqlite_module` might get changed in the upstream, so added some padding space for our custom functions
* generate ffi bindings
* Add a github workflow to test crsqlite
* fix crsqlite: remove `pLibsqlModule` references
* Add tests for sqlite-vss extension
* wal.c: fix the function pointer type
A 100% benign error, because the function has the correct signature,
but gets rid of a compiler warning.
* regenerate sqlite3.c amalgamation
* libsql-ffi: regenerate SQLite3MultipleCiphers' sqlite3.c
... not needed later, because an existing patch series switches it
to use the sqlite3.c amalgamation file from bundled/src/,
but let's update it for now.
We need the `sqlite3_get_autocomit()` helper in the TypeScript SDK, for
example. The fact that it's not exported seems to be just an omission so
let's add it.
Based on https://gist.github.com/psarna/0dbd95424fb7121bb54bf5f961e1e117
from SQLite3MultipleCiphers, let's try and make libsql flexible enough
to not require patching the amalgamation files at all in order to integrate
with SQLite3Multiple ciphers.
For starters, their codec functions are instead declared as libsql_pager_codec
and libsql_pager_has_codec. The idea for future integration is that
if an appropriate compilation flag is on, we assume that the code will be compiled
with additional symbols that provide all the implementation we need, e.g.
libsql_pager_codec_impl() will just transiently call sqlite3mcPagerCodec().
If the following directives are defined:
- LIBSQL_PRE_VFS_HOOK
- LIBSQL_EXTRA_PRAGMAS
- LIBSQL_EXTRA_URI_PARAMS
, a few new symbols need to be compiled in, respectively:
- libsql_pre_vfs_hook(const char *)
- libsql_extra_pragma(sqlite3 *, const char *, void *)
- libsql_handle_extra_uri_params(sqlite3 *, const char *),
libsql_handle_extra_attach_params(sqlite3 *, const char *,
const char *, sqlite3_value *, char **)
Those hooks, combined with virtual WAL, should be plenty enough to
integrate SQLite3MultipleCiphers without patching sqlite3.c code.
Those are really weird internally, and assume that the db path string
you pass as an argument resides in a very specific place in memory,
preceded by 4 zero bytes, and followed by journal and wal names.
The WALv2 rework broke this magic assumption, so we hereby restore it.
As follow-up, we should move this filename management to virtual WAL,
because with virtual WAL implementations you sometimes don't even have
a notion of "WAL file path".
Refactor of the virtual wal API. Drop global wals in favor of passing
the wal implementation as an argument to libsql_open.
The WAL interface is split in two sets of virtual methods:
- `create_wal` is passed when opening a sqlite connection. It's role is
to instantiate a wal.
- `libql_wal` is the wal itself, created by the `create_wal`.
The sqlite3_wal and `sqlite3_create_wal` implementation are completely
decoupled from the wal implementation. They are the default
implementation when using the traditional open methods, and
sqlite3_create_wal is exposed as a global variable, and can be composed
with other wal implementations.
On top of a special VFS, `make wasi` now also compiles-in
a specialized virtual WAL methods module. It doesn't do anything
right now (just falls back to the regular implementation),
but can be used to call back to the host, inject hooks, etc.
