Currently the codepath used when rusqlite is disabled, is
Unix-platform-dependent.
For instance, here is the error we compiling from Windows:
```
error[E0433]: failed to resolve: could not find `unix` in `os`
--> C:\Users\runneradmin\.cargo\git\checkouts\libsql-311658d335deb3b1\9b04f1d\libsql-sys\src\connection.rs:276:26
|
276 | use std::os::unix::ffi::OsStrExt;
| ^^^^ could not find `unix` in `os`
|
note: found an item that was configured out
--> /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\std\src\os\mod.rs:27:9
note: the item is gated here
--> /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\std\src\os\mod.rs:19:1
note: found an item that was configured out
--> /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\std\src\os\mod.rs:65:9
note: the item is gated here
--> /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\std\src\os\mod.rs:64:1
error[E0599]: no method named `as_bytes` found for reference `&OsStr` in the current scope
--> C:\Users\runneradmin\.cargo\git\checkouts\libsql-311658d335deb3b1\9b04f1d\libsql-sys\src\connection.rs:277:73
|
277 | let path = std::ffi::CString::new(path.as_ref().as_os_str().as_bytes())
| ^^^^^^^^
|
help: there is a method `as_encoded_bytes` with a similar name
|
277 | let path = std::ffi::CString::new(path.as_ref().as_os_str().as_encoded_bytes())
| ~~~~~~~~~~~~~~~~
Some errors have detailed explanations: E0433, E0599.
For more information about an error, try `rustc --explain E0433`.
error: could not compile `libsql-sys` (lib) due to 2 previous errors
```
This patch is adding a platform-independent branch.
The database path is expected to be UTF-8 by the libsql native library.
However, in the real world, all Unix paths are not necessarily UTF-8.
For this reason, I understand why the Unix codepath is attempting a
direct conversion from OsString to CString.
However, it’s not possible to do something similar on other platforms.
For these platforms, we are enforcing correct UTF-8 using `to_str`. At
least, it should work reasonably well in most cases.
Currently the codepath used when rusqlite is disabled, is
Unix-platform-dependent. This patch is adding a platform-independent
branch. The database path is expected to be UTF-8 by the libsql native
library. However, in the real world, all Unix paths are not necessarily
UTF-8. For this reason, I understand why the Unix codepath is attempting
a direct conversion from OsString to CString. However, it’s not possible
to do something similar on other platforms. For these platforms, we
are enforcing correct UTF-8 using to_str. At least, it should work
reasonably well in most cases.
A common complain with libSQL is how to run extensions. The main
mechanism, with a .so, has a lot of issues around how those .so are
distributed.
The most common extensions are the ones in the sqlean package. We can
improve this experience by bundling them in our sqlite build.
Not all SQLean extensions are kosher: some of them, like fileio, use
the vfs. Others, are deemed too complex.
The extensions included here are a subset that we deem important enough,
and low risk enough, to just be a part of the main bundle.
This adds a new xReadFrameRaw() function to the virtual WAL API, which
upper layers can use to fetch the full frame, including the page number,
that is useful for appending frames to a WAL.
* Create a drop in replacement for `Either` with `Any`
* generate Eitherxx variants
* generate Eitherxx variants
---------
Co-authored-by: ad hoc <postma.marin@protonmail.com>
* introduce connection manager
* remove unused wal methods
* remove lock stealer
* Make use of ConnectionManager in LibsqlConnection
it now takes a W: WalWrap instead of a WalManager. This is because we
want to inject the connection manager at the bottom of the wal wrapping
chain.
* add missing deps
* turn ReplicationLogger into a WrapWal
* update spots to to pass wal wrapper instead of wal manager
* remove dbg
* fmt
* fix sqlite3 rust tests
* libsql: Make encryption cipher configurable
Introduce a `EncryptionConfig` struct to configure both encrytion cipher
and key. Needed to support multiple ciphers.
Fixes#951
* libsql-ffi: Switch to SQLCipher as the default cipher
Fixes#893
* namespace,replication: add LogFile encryption
Anything that uses our LogFile format can now be encrypted
on-disk.
Tested locally by seeing that `wallog` file contains garbage
and no sensible plaintext strings can be extracted from it.
* test fixups
* libsql-ffi: add libsql_generate_initial_vector and...
... libsql_generate_aes256_key to make them reachable from Rust.
* connection: expose additional encryption symbols
* libsql-server: derive aes256 from user passphrase properly
And by properly, I mean calling back to SQLite3MultipleCiphers' code.
* replication: rename Encryptor to FrameEncryptor
Encryptor sounds a little too generic for this specific use case.
* replication: add snapshot encryption
It uses the same mechanism as wallog encryption, now abstracted
away to libsql-replication crate to be reused.
* replication: add an encryption feature for compilation
* cargo fmt pass
* fix remaining SnapshotFile::open calls in tests
* logger: add an encryption test
* replication: use a single buffer for encryption
Ideally we could even encrypt in place, but WalPage is also
used in snapshots and it's buffered, and that makes it exceptionally
annoying to explain to the borrow checker.
* bottomless: restore with libsql_replication::injector
... instead of the transaction page cache. That gives us free
encryption, since the injector is encryption-aware.
This patch doesn't hook encryption_key parameter yet, it will
come in the next patch.
* bottomless: pass the encryption key in options
For WAL restoration, but also to be able to encrypt data that gets
sent to S3.
* bottomless: inherit encryption key from db config if not specified
* libsql-sys: add db_change_counter()
The helper function calls the underlying C API to extract
4 bytes from offset 24 of the database header and return it.
It's the database change counter, which we can use to compare
two databases and decide which one is newer than the other.
* bottomless: use sqlite API to read database metadata
With encryption enabled, we can no longer just go ahead and read data
from given offsets, we must go through the VFS layer instead.
Fortunately, we can just open a database connection and ask for all
the metadata we need.
* libsql-sys: make db change counter actually read from the db file
* bottomless: treat change counter == 1 as a new database
... which it is, after setting the journal mode. Otherwise we decide
too eagerly that the local database is the source of truth.
* libsql-server: fix a local embedded replica test
rebase conflict with encryption
* bottomless-cli: allow passing the encryption key
* replication: rebase new test to the new api
* snapshots: do not try to decrypt headers
They are not encrypted, so we shouldn't attempt to decrypt the data.
* logger: restore encrypted frames during recovery
Instead of decrypting and encrypting back, we just copy encrypted
frames as is during the recovery process, saves IO.
* compaction: clear unused encryption_key parameter
It wasn't used since for compaction we only need headers,
which are unencrypted.
* replication: switch to FrameBorrowed::new_zeroed
Following MarinPostma's suggestion.
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
* replication: rebase chores, fixing parameters
* libsql-replication: use page_mut() to decrypt data in-place
* rustfmt
* bottomless: use 0 for disabling autocheckpoint
... instead of u32::MAX. Effectively it's similar, but 0 is the correct
choice.
* rustfmt
* libsql-server: make cbc, aes optional for encryption only
* post-rebase fixes
* libsql-replication: suppress warnings when no encryption
* libsql: add encryption support for local databases
* libsql: add bytes dependency for encryption
* libsql-ffi: build libsqlite3mc without debug symbols
Technically it should just depend on cargo build mode,
but that's left for a follow-up.
* bindings: an attempt to compile bindings with releasemode
... partially to save space, but also to make them faster.
---------
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
- add the checkpoint callback to Wal::checkpoint
- use dynamic dispatch for callbacks (correctness issue)
- pass `frames_in_wal` and `backfilled` as ref to `Wal::checkpoint`
because sqlite can set them despite returning `SQLITE_BUSY`
