0
0
mirror of https://github.com/tursodatabase/libsql.git synced 2025-07-23 14:31:47 +00:00
Commit Graph

1592 Commits

Author SHA1 Message Date
ad hoc
965f3dc7d7 per-namespace sentinel (#1202) 2024-03-13 11:37:33 +00:00
Bartosz Sypytkowski
3de73ef356 Namespace::destroy: add parameter to perform soft delete (preserve backup) (#1204)
* Namespace::destroy: add parameter to perform soft delete (preserve backup)

* make sure that log statement won't panic
2024-03-13 11:14:45 +00:00
ad hoc
0d5dc18ce1 rename migration tables (#1200)
* rename __libsql_migration_tasks to sqlite3_libsql_tasks

* rename job tables
2024-03-13 11:03:55 +00:00
ad hoc
b9ab41c485 ignore legacy auth when new auth is present (#1192) 2024-03-13 06:30:53 +00:00
ad hoc
c79cec3ee2 move migration api to user api (#1198) 2024-03-12 20:09:07 +00:00
ad hoc
a95bdf4d35 job status notifier & wait for dry run success (#1197) 2024-03-12 19:55:41 +00:00
ad hoc
153c8382d8 remove broken test (#1199) 2024-03-12 18:00:30 +00:00
Bartosz Sypytkowski
d945d41566 Hrana: don't close the stream at the end of prepared statement execution (#1185)
* Hrana: don't close the stream at the end of prepared statement execution

* hrana stream and transaction: check for tokio runtime existence on drop

* standalone::execute_transaction test: raise simulation duration

* hrana: added comments to conditions for stream closing

* hrana: added comments to conditions for stream closing
2024-03-12 11:57:07 +00:00
Pekka Enberg
969560c6d4 libsql-server: s/conccurent/concurrent/ (#1194) 2024-03-12 11:01:55 +00:00
ad hoc
e388a72f2d implement missing function on schema db (#1193) 2024-03-12 09:47:02 +00:00
ad hoc
5bad81bc11 block migration during db operation (#1183)
* introduce schema lock

* synchronize migration with db creation
2024-03-11 17:25:10 +00:00
ad hoc
fe2ebffe28 check schema migration auth (#1191) 2024-03-11 17:09:43 +00:00
ad hoc
5ece28f4fe gid tag token claim (#1190) 2024-03-11 16:14:52 +00:00
ad hoc
95cb5b0b5c prevent ns deletion during migration (#1184) 2024-03-11 15:42:41 +00:00
ad hoc
1528128fce allow ddl override (#1187) 2024-03-11 15:00:26 +00:00
ad hoc
833350dae5 handle scheduler errors (#1175)
* handle scheduler errors

* fix query
2024-03-09 11:40:31 +00:00
ad hoc
998d330428 extend test timeout (#1182) 2024-03-09 11:39:39 +00:00
Athos
9092268a89 Report stats latencies with microseconds accuracy (#1174) 2024-03-08 17:04:26 +00:00
Avinash Sajjanshetty
226fba9a12 Add tests for data migration in schema dbs (#1176)
- Added a test which performs data migration including
  inserts, updates and deletes
- Added a test which shows that schema migration fails
  if there is any conflicting data in the linked dbs
2024-03-08 16:11:26 +00:00
Avinash Sajjanshetty
ae708f8952 Fork a new DB when creating from shared schema (#1154)
* Fork a new DB when creating from shared schema

* Use the correct db type to get the wal logger

* Add a test for a new db in shared schema

- When a new db is created with a shared schema, it should have
  the same schema and the schema version
- When a new migration is done, the new db and all the existing
  dbs should have the same schema and version

* Move schema db fork logic inside namespace mod

* clean up and remove unnecessary store call
2024-03-08 14:32:22 +00:00
ad hoc
21fa8076e9 fix flaky shared schema test (#1173) 2024-03-08 13:05:19 +00:00
ad hoc
c9ba69245e Handle dry run failure, and abort migration (#1163)
* test snasphots

* add test

* handle dry run failure, and abort all tasks

* fmt
2024-03-08 11:01:32 +00:00
ad hoc
b67550ffea fix conccurency bug in test (#1164) 2024-03-08 09:16:02 +00:00
Avinash Sajjanshetty
73909accf6 Update tests to check if all schema versions are same (#1170) 2024-03-08 08:58:27 +00:00
Piotr Jastrzębski
fef7689301 Make sure setup_migration_table does not change schema_version (#1161)
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
2024-03-07 17:15:23 +00:00
Avinash Sajjanshetty
02bd31c69d Disable forking for a shared schema database (#1160) 2024-03-07 14:48:59 +00:00
Avinash Sajjanshetty
baecc81789 Add optional flush param when storing in MetaStore (#1159)
Added a new method `store_and_maybe_flush` in MetaStore, which gives you an option to flush the config changes to disk.

The previous `store` method behaviour is not changed. It internally calls `store_and_maybe_flush` method, stores the config and flushes to disk.

This patch fixes a bug in fork. Earlier, we were flushing the target config before actually creating a fork. If the fork failed, MetaStore would have invalid data. Now, we store the config, without flush, create the fork. Iff successful, flush the target config to disk.
2024-03-07 15:21:26 +01:00
Piotr Jastrzębski
9b96dbc349 Cleanup errors (#1158)
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
2024-03-07 12:29:53 +00:00
Lucio Franco
405f605705 libsql: prevent database overwrite misuse (#1153)
Closes #1145
2024-03-07 11:59:46 +00:00
Bartosz Sypytkowski
25bd951ee1 Schema migrations metastore backup (#1144)
backup meta store on state change and schema on update

Co-authored-by: ad hoc <postma.marin@protonmail.com>
2024-03-07 11:35:27 +00:00
Avinash Sajjanshetty
0c1c35f8bf Disallow creation of shared schema db with a dump (#1156)
Disallow shared schema db with dump
2024-03-07 10:58:53 +00:00
ad hoc
bf19f3e1d7 reject migration containing migration statements (#1149) 2024-03-06 18:12:45 +00:00
Bartosz Sypytkowski
266ab4f0e4 adding backup sync await for schema migrations (#1140)
* adding backup sync await for schema migrations

* move backup_sync call after write_block was set

* perform backup in task

* save shared schema db backup once all databases confirm dry run

* added schema job backup sync

* adjusted tests

* ensure that there are no ongoing transactions before migrated db backup

* fixed flaky test

* tweaks

---------

Co-authored-by: ad hoc <postma.marin@protonmail.com>
2024-03-06 17:21:29 +00:00
ad hoc
75321d56d3 use custom query builder to collect migration error (#1146)
use custom query builder to collect migration error
2024-03-06 15:31:46 +00:00
ad hoc
23f8fd2711 prevent namespace creation/update when there is a pending migration on shared schema (#1137) 2024-03-05 22:18:22 +00:00
Piotr Jastrzębski
8f897d510e Block writes during schema migration (#1139)
* Add block_writes to Connection

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Add block_writes to PrimaryDatabase

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Use block_writes in Connection::run

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Unblock writes once migration is finished

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Block writes on startup if there are pending migration tasks

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Block writes when a migration task is enqueued

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* twix

---------

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
Co-authored-by: ad hoc <postma.marin@protonmail.com>
2024-03-05 22:17:36 +00:00
Pekka Enberg
4e0ce557a8 libsql: Rename periodic_sync to sync_interval (#1125)
Rename the configuration option for periodic sync to `sync_interval` to
align with what we're doing with the rest of the SDKs.
2024-03-05 19:05:46 +00:00
ad hoc
2c1b2d0f48 handle job retry (#1136)
do not re-perform task if already performed
2024-03-05 17:41:32 +00:00
Bartosz Sypytkowski
bb1dd03748 Migrations status admin api (#1134)
* added an endpoint to ask for a general migration status

* return detailed information about individual migration job progress

* integrated migrations API in tests

* post-rebase

* adjusted code after rebase
2024-03-05 16:20:30 +00:00
Pekka Enberg
6b13d77519 libsql: Deasyncify some Conn trait methods (#1135)
There's no reason to have `is_autocommit()`, `changes()` or
`last_insert_row()` methods async because they're all fully local and
cannot block. Deasyncify them.
2024-03-05 13:44:53 +00:00
ad hoc
50648a6b4b prevent task enqueue when pending task (#1131)
check for already enqueued job for schema + migrate schema db ops to
schema module.
2024-03-05 10:31:06 +00:00
ad hoc
7de9b48110 trigger delete enqueued tasks (#1126)
create a trigger to remove tasks from enqueued task table on update
2024-03-04 16:42:06 +00:00
Piotr Jastrzębski
1826582133 Embedded replicas encryption test (#1124)
* Embedded replicas encryption test

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Fix encryption embedded replica test

* ci: add encryption tests

* add clean cipher build dir

---------

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
Co-authored-by: Lucio Franco <luciofranco14@gmail.com>
2024-03-04 16:09:02 +00:00
ad hoc
2b06b6b7d8 schema migration (#1110)
happy path schema migration
2024-03-04 13:15:29 +00:00
Lucio Franco
9dbf604843 libsql: prefix client_wal_index file (#1104)
* libsql: prefix client_wal_index file

* remove create_dir_all
2024-03-04 12:43:39 +00:00
Bartosz Sypytkowski
aa2490cd36 Bottomless savepoint (#1085)
* bottomless: added ability to wait for uploaded frames without shutting down

* bottomless: savepoint tracker

* expose bottomless backup savepoint through PrimaryDatabase API

* tests for detached progression updates in bottomless progression tracker

* bottomless: reset savepoint tracker on new generation

* bottomless: expose savepoint tracker on PrimaryDatabase
2024-03-04 09:51:48 +00:00
Athos
7f9e289681 Aggregated stats for queries with most elapsed_time (#1086)
* Aggregated stats for queries with most elapsed_time

* Add query percentile support to stats (#1112)

* Record query latency percentiles with hdrhistogram

* Expose queries percentiles through stats response

* Simplify QueriesStats transformation to QueriesStatsResponse

* Reset query stats at the beginning of every hour (#1118)

* Add expires_at to control when QueriesStats should be reset

* Add query count and elapsed sum to query stats response

Also group all latency aggregations together under the elapsed key

* Remove option sprawl on QueriesStats fields

By making the whole struct optional where it is used

* Add created_at to stats queries responde object

* Set queries stats to none when stats is created

This will make the API response the same when no queries
have been recorded on the queries stats struct. This can
happen right after:
- the stats struct initialization
- the stats queries object expiration

Example of stats API response when no queries have been
recorded:

{
  ...
  "queries": null
}

* Use if else instead of early return from stats to response
2024-03-01 21:43:50 +00:00
Avinash Sajjanshetty
612aae4277 server: trim quotes in namespace string in attach statement (#1117)
* Add a regression test when UUIDs are used in attach
statements

Also, added a test to check if the UUID is wrapped
in quotes

* bugfix: trim namespace string if it is wrapped in quotes

the client might send namespace wrapped in quotes in case of uuids,
so we will just trim the string.
2024-03-01 15:08:01 +00:00
ad hoc
169ef7fa11 Factorize program execution in VM object (#1109)
factorize program Execution in Vm
2024-03-01 10:44:24 +00:00
ad hoc
adf36104ad fix bad attach name (#1113)
* add test

* fix auth using attach alias instead of target
2024-02-29 14:15:43 +00:00