* update WIP list
* prefer local max_frame_no
this assumes that cache is always up to date and uses that instead
of calling the storage server. later we will add replication and make
sure that cache is upto date.
* Prefer local reads and also save page versions locally
Let's store `page_no` along with the `frame_no`. And then while reading,
we will do a range read, read frame which is less than txn's
`max_frame_no`
* minor improvements: index creation, fix max_frame_num query
* update local cache on inserts
* avoid unnecessary call to ss in `find_frame`
* remove unused `get_frame`
* eliminiate all calls to storage server during read path
* Make PRAGMA synchronous configurable for a Namespace
SQLite allows setting synchronous values to any four levels. This
patch provides an option to set the level for a namespace which will
be used for every connection. It defaults to `NORMAL`.
allowed values: https://www.sqlite.org/pragma.html#pragma_synchronous
e.g. to enable:
curl -X POST http://sqld-admin-endpoint/v1/namespaces/<namespace>/config -H "Content-Type: application/json" -d '{"durability_mode": "strong"}' -v
* Make `durability_mode` field optional in proto
* Set `default` in the enum
Co-authored-by: ad hoc <postma.marin@protonmail.com>
---------
Co-authored-by: ad hoc <postma.marin@protonmail.com>
* cancel previous running actions if new commits are pushed
* cancel-in-progress for rust tests
* cancel-in-progress for c-bindings workflow
* cancel-in-progress for test GitHub actions
* add basic functions for working with f32/f64 vectors
* add header file
* fix comment style
* add generic code for working with vectors
* register vector functions
* fix edge cases in vector extract impl
* add static flag for vector (will be used later)
* add basic TCL-based tests
* guard vector functions with SQLITE_OMIT_VECTOR
* adjust build scripts
* delete libsql_vector_idx from this branch
* add one more test
* update bundles
* fix asserts
* fix constants (#define-s are universal but const int doesn't recognized by all compilers)
* update bundles
* review fixes
* build bundles
* add comment about linking math library
* add alter column test
* apply alter column only to the table and ignore indices/triggers/views etc
* fix tests
* update bundle
* add test in rust_suite
* add basic TCL tests for ALTER COLUMN libsql feature
* propagate length of the new column definition directly from the parser
- this will allow libsql to automatically handle comment and space characters appended to the column definition
* small formatting fixes
* cargo xtask build-bundled
* add test against sqlite3 source in rust_suite
* fix test in rust_suite
It is important to verify that the data we got from Foundation DB matches with what we
want, since we are doing a range query.
for example, say we searched for ('db_name42', 10, 20). If this page does not exist, then
it could match with ('db_name41', 10, 20) or with ('db_name42', 9, 20) since it does
lexicographic search.
- Changed transaction table to use `(txn_id, page_no)` as composite
primary key
- Updated `insert_frame` to ignore unique constraint violation since
frames are immutable
- Change `insert_page` method to upsert
* Add basic tonic error handling
Added a simple error to handle `WriteConflict`s
* Improve the way we handle Foundation DB keys
Foundation DB keys support tuples. Earlier we used plain strings. For
example to store page 5, which had frame no 8, we did `ns/f/5/8`.
However this breaks lexicographic sorting. For e.g. we want `ns/f/5/8`
to be smaller than `ns/f/5/18`, but since we store as a plain string
the comparison breaks.
So we let FDB handle encoding with the `pack` API which encodes the
key properly as tuple `(ns, f, 5, 8)`
* Send `max_frame_no` while doing inserts and find frame
Whenever a txn is started, we keep the current `max_frame_no` in the
transaction's state. For the subsequent find and insert requests, we
send this value to storage server.
For a read txn, it should not read any frames greater than it's
max_frame_no. For a write txn, if the max_frame_no has changed
meanwhile, then txn should be aborted.
* Allow multiple writers at the storage server
This patch has two important changes:
- Storage server now uses `max_frame_no` from the txn to read frames
It will make sure to not read any frames beyond the `max_frame_no`
of the transaction. This fixes the bug in isolation level.
- Storage server now checks `max_frame_no` before inserting. If it
has changed meanwhile, then it rejects the insertion request.
This lets us have multiple (non concurrent) writers or even multiple
logical primary instances.
* minor cargo fmt fixes
* replace in-memory cache with local sqlite db
Previously, `libsql-storage` stored all the data in-memory. It kept a
in-memory frame cache and also cached all the transient transaction
writes in-memory. This patch changes it to store the data on local disk.
This data is transient in nature, disk loss should not cause any issues.
* fix: remove unnecessary `namespace` column in local cache
Each namespace gets its own local cache. So there is no need of
namespace column at all.
* Add some explainer about how we use the local cache
We use LocalCache to cache frames and transaction state. Each namespace gets its own cache
which is currently stored in a SQLite DB file, along with the main database file.
Frames Cache:
Frames are immutable. So we can cache all the frames locally, and it does not require them
to be fetched from the storage server. We cache the frame data with frame_no being the key.
Transaction State:
Whenever a transaction reads any pages from storage server, we cache them in the transaction
state. Since we want to provide a consistent view of the database, for the next reads we can
serve the pages from the cache. Any writes a transaction makes are cached too. At the time of
commit they are removed from the cache and sent to the storage server.
The `execute()` method returns an error if the query returns rows, which
makes it unsuitable for running any type of SQL queries. Similarly, the
`query()` method only works on SQL statements that return rows.
Therefore, the `run()` method is provided to execute any type of SQL
statement.
This is needed, for example, for the libSQL JavaScript library that
needs to preserve `better-sqlite3` semantics.
