* Hrana: don't close the stream at the end of prepared statement execution
* hrana stream and transaction: check for tokio runtime existence on drop
* standalone::execute_transaction test: raise simulation duration
* hrana: added comments to conditions for stream closing
* hrana: added comments to conditions for stream closing
- Added a test which performs data migration including
inserts, updates and deletes
- Added a test which shows that schema migration fails
if there is any conflicting data in the linked dbs
* Fork a new DB when creating from shared schema
* Use the correct db type to get the wal logger
* Add a test for a new db in shared schema
- When a new db is created with a shared schema, it should have
the same schema and the schema version
- When a new migration is done, the new db and all the existing
dbs should have the same schema and version
* Move schema db fork logic inside namespace mod
* clean up and remove unnecessary store call
* Add block_writes to Connection
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* Add block_writes to PrimaryDatabase
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* Use block_writes in Connection::run
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* Unblock writes once migration is finished
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* Block writes on startup if there are pending migration tasks
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* Block writes when a migration task is enqueued
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
* twix
---------
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
Co-authored-by: ad hoc <postma.marin@protonmail.com>
* added an endpoint to ask for a general migration status
* return detailed information about individual migration job progress
* integrated migrations API in tests
* post-rebase
* adjusted code after rebase
There's no reason to have `is_autocommit()`, `changes()` or
`last_insert_row()` methods async because they're all fully local and
cannot block. Deasyncify them.
* Add a regression test when UUIDs are used in attach
statements
Also, added a test to check if the UUID is wrapped
in quotes
* bugfix: trim namespace string if it is wrapped in quotes
the client might send namespace wrapped in quotes in case of uuids,
so we will just trim the string.
* test snapshots
* allow multi-scope auth
* hors-sujet: fix C compile warning
* update jwt and add hashbrown feature
* replace Auth with RequestContext
* extract RequestContext
* fix auth check for multi-scopes
* pass NamespaceName in StmtKind::Attach
* fallback to global auth if key is not set in ns config
* add allow_attach to namespace creation
* return meta store from namespace store
* test attach
* fmt
* review edits
* libsql: attach databases from other namespaces as readonly
With this proof-of-concept patch, other namespaces hosted
on the same sqld machine can now be attached in readonly mode,
so that users can read from other databases when connected
to a particular one.
* connection: add allow_attach to config
Default is false, which means connections are blocked from attaching
databases. If allowed, colocated databases can be attached in readonly
mode.
Example:
→ attach another as another; select * from another.sqlite_master;
TYPE NAME TBL NAME ROOTPAGE SQL
table t3 t3 2 CREATE TABLE t3(id)
* libsql,namespaces: add client-side ATTACH support
* attach: support ATTACH x AS y aliasing
We're going to need it, because the internal database names in sqld
are uuids, and we don't expect users to know or use them.
* attach: fix quoted db names
In libsql-server, raw db names are uuids that need to be quoted,
so that needs to be supported in the ATTACH layer.
As a bonus, "names" that are actually file system paths are refused
to prevent abuse.
* libsql-server: drop stray serde(default) from allow_attach
* libsql-replication: update proto files
* libsql-replication: regenerate protobuf
* tests: move attach to its own test
* libsql-replication: fix proto number after rebase
* namespace,replication: add LogFile encryption
Anything that uses our LogFile format can now be encrypted
on-disk.
Tested locally by seeing that `wallog` file contains garbage
and no sensible plaintext strings can be extracted from it.
* test fixups
* libsql-ffi: add libsql_generate_initial_vector and...
... libsql_generate_aes256_key to make them reachable from Rust.
* connection: expose additional encryption symbols
* libsql-server: derive aes256 from user passphrase properly
And by properly, I mean calling back to SQLite3MultipleCiphers' code.
* replication: rename Encryptor to FrameEncryptor
Encryptor sounds a little too generic for this specific use case.
* replication: add snapshot encryption
It uses the same mechanism as wallog encryption, now abstracted
away to libsql-replication crate to be reused.
* replication: add an encryption feature for compilation
* cargo fmt pass
* fix remaining SnapshotFile::open calls in tests
* logger: add an encryption test
* replication: use a single buffer for encryption
Ideally we could even encrypt in place, but WalPage is also
used in snapshots and it's buffered, and that makes it exceptionally
annoying to explain to the borrow checker.
* bottomless: restore with libsql_replication::injector
... instead of the transaction page cache. That gives us free
encryption, since the injector is encryption-aware.
This patch doesn't hook encryption_key parameter yet, it will
come in the next patch.
* bottomless: pass the encryption key in options
For WAL restoration, but also to be able to encrypt data that gets
sent to S3.
* bottomless: inherit encryption key from db config if not specified
* libsql-sys: add db_change_counter()
The helper function calls the underlying C API to extract
4 bytes from offset 24 of the database header and return it.
It's the database change counter, which we can use to compare
two databases and decide which one is newer than the other.
* bottomless: use sqlite API to read database metadata
With encryption enabled, we can no longer just go ahead and read data
from given offsets, we must go through the VFS layer instead.
Fortunately, we can just open a database connection and ask for all
the metadata we need.
* libsql-sys: make db change counter actually read from the db file
* bottomless: treat change counter == 1 as a new database
... which it is, after setting the journal mode. Otherwise we decide
too eagerly that the local database is the source of truth.
* libsql-server: fix a local embedded replica test
rebase conflict with encryption
* bottomless-cli: allow passing the encryption key
* replication: rebase new test to the new api
* snapshots: do not try to decrypt headers
They are not encrypted, so we shouldn't attempt to decrypt the data.
* logger: restore encrypted frames during recovery
Instead of decrypting and encrypting back, we just copy encrypted
frames as is during the recovery process, saves IO.
* compaction: clear unused encryption_key parameter
It wasn't used since for compaction we only need headers,
which are unencrypted.
* replication: switch to FrameBorrowed::new_zeroed
Following MarinPostma's suggestion.
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
* replication: rebase chores, fixing parameters
* libsql-replication: use page_mut() to decrypt data in-place
* rustfmt
* bottomless: use 0 for disabling autocheckpoint
... instead of u32::MAX. Effectively it's similar, but 0 is the correct
choice.
* rustfmt
* libsql-server: make cbc, aes optional for encryption only
* post-rebase fixes
* libsql-replication: suppress warnings when no encryption
* libsql: add encryption support for local databases
* libsql: add bytes dependency for encryption
* libsql-ffi: build libsqlite3mc without debug symbols
Technically it should just depend on cargo build mode,
but that's left for a follow-up.
* bindings: an attempt to compile bindings with releasemode
... partially to save space, but also to make them faster.
---------
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
This adds a new periodic sync configuration item to the `Builder` that
will allow users to set a duration at which they want the background
sync to sync.
This adds a new `Builder` type that can now be used to construct the
`Database` type. This will scale better as we add more varied options.
This commit also deprecates the old builder types and will produce a
warning that will push users to using the new `Builder` type. This will
then allow us to remove the old deprecated constructors at some point in
the future.
