JavaScript promise rejection: Loading CSS chunk index-domready failed. (error: https://sirherobrine23.com.br/assets/css/index-domready.9de057c0.css). Open browser console to see more details.
0
0
mirror of https://github.com/tursodatabase/libsql.git synced 2025-07-09 07:49:24 +00:00
Commit Graph

1109 Commits

Author SHA1 Message Date
d7b7647159 Merge pull request #480 from tursodatabase/bump-tungstenite
bump tungstenite
2023-10-19 16:12:33 +00:00
9e906cf965 bump tungstenite 2023-10-19 17:36:20 +02:00
f15800793a re-enable embedded replica tests 2023-10-19 09:11:25 +02:00
9c90ccffa5 clean repo 2023-10-18 13:49:57 +02:00
fa3411976e fix server tests 2023-10-17 22:22:37 +00:00
87a2eef19e remove useless files 2023-10-17 17:41:48 +02:00
99996d0cfe move sqld release workflow 2023-10-17 17:41:29 +02:00
71ca43a7b6 make sqld compile 2023-10-17 17:41:26 +02:00
76e88487c2 sqld: turn off wasm functions by default
It also transiently fixes the "malformed db" issue with embedded
replicas. Those replicas didn't create a "libsql_wasm_func_table"
when first connecting, and sqld did, due to having wasm functions
enabled.
But, since our sqld parser still doesn't accept CREATE FUNCTION
yet, there's no harm in disabling the support.
2023-10-17 16:33:27 +02:00
e2aeecd42d sqld: honor checkpoint-interval-s even without bottomless (#781)
There's no reason I'm aware of to only make checkpoint-interval-s
option effective if bottomless is enabled. Botomlessless instances
should also be able to checkpoint periodically.
2023-10-16 16:58:43 +02:00
85b3358a4d Revert "Same-thread frame injection (#695)" (#775)
This reverts commit 61b0406dd6.
2023-10-16 08:04:05 +00:00
8652fe6312 libsql: vacuum less often (#778)
For specific edge cases, a 32MiB threshold is a little small.
Let's bump it to at least 256MiB of data and inform about vacuuming
with info log level.
2023-10-16 06:15:37 +00:00
7cf962c700 error: fix typo and normalize messages with a Title starting word (#773) 2023-10-13 17:49:06 +00:00
2417e6f9c4 sqld: Add extended_code support (#768)
* Add embedded replica test

* sqld: Add `extended_code` support
2023-10-13 17:43:57 +00:00
dba4782aff fix various comment, code var and doc typos (#774)
* connection: fix various comment, code var and doc typos

* replication: fix various comment, code var and doc typos

* namespace: fix various comment, code var and doc typos

* metrics: fix typo in metric docstring

* hrana/stmt: fix code typo verion -> version

* query_result_builder: fix code typo weigths -> weights

* h2c: fix typo in docstring

* query_analysis: fix typo in docstring
2023-10-13 17:39:35 +00:00
73b92038c3 Update README.md (#772) 2023-10-13 15:48:08 +00:00
e855161ee9 docs: add a wildcard domain mention to the user guide (#771)
While testing multitenancy, it's more convenient to use a wildcard
domain rather than edit /etc/hosts manually.
2023-10-13 11:56:10 +02:00
61b0406dd6 Same-thread frame injection (#695)
* add same thread injector

* refactor Frame API; primary return commit frame for snapshots

* hook new injector

* add test assets

* handle fatal replication error

* use persistent replication table

* add doc commen to snapshot frames_iter_from

* review fixes

* fmt

* handle potential injector execute error
2023-10-13 07:49:00 +00:00
6cc36c740d Make cargo-chef effective for CI builds (#763)
* Optimize runtime container layers

* Use Github Actions cache for Docker

* Move rm -rf /var/lib/apt/lists/* to its own line
2023-10-13 07:29:06 +00:00
e83af252dc disable metrics for tests (#769) 2023-10-13 07:25:22 +00:00
931275567d prom metrics (#760)
* wip

* add prometheus metrics probes

* expose metrics admin endpoint

* move metric route to /metrics

* use histogram macro
2023-10-12 13:50:36 +00:00
81bdd501a6 use anyhow bail to return error (#756) 2023-10-11 15:37:42 +00:00
6f841a621e sqld: make wasm opt-out (#758)
This will allow ARMv7 architecture users to compile sqld.
Wasmtime currently does not have support for that arch.
2023-10-11 15:37:16 +00:00
c685bb697e Make CI faster (#754)
* Group all checks together in the same workflow

Reusing artifacts ends up being faster than parallelizing with the
current CI resources.

* Set same RUSTFLAGS in both jobs && remove unnecessary steps

* Cache rust artifacts
2023-10-11 15:35:33 +00:00
57ba103d4c verify after db restore is done (#751) 2023-10-10 08:57:46 +00:00
7f0fdcfaaa bump sqld to v0.21.9 (#753) 2023-10-09 16:20:39 +00:00
528b3e2c52 sqld: Revert be7386 and remove decoding limits (#752)
This reverts be7386 and removes decoding limits for the `ReplicationLogProxyClient`
so that when replication requests get forwarded to the primary and it returns a full
batch of `1024` frames it does not trigger the 4mb decoding limit set by default for
tonic gRPC clients.
2023-10-09 16:20:27 +00:00
4b109feb4e add retry to s3 uploads in bottomles (#750) 2023-10-09 13:38:17 +02:00
8ff275cf3d enable savepoint (#472)
* add savepoint and release stmt categories

* forbid savepoints in legacy http
2023-10-09 09:20:10 +00:00
cd80e60102 bump sqld to v0.21.8 (#749) 2023-10-06 19:49:22 +00:00
1e2d7d16f4 hrana: Properly catch stmt errors in batch (#748) 2023-10-06 19:40:37 +00:00
2b7b2d4eee Remove encoding message limit (#747) 2023-10-06 15:41:29 +00:00
22fad81348 remove useless docker build in CI (#746) 2023-10-06 13:59:54 +00:00
0a643bde74 Remove context calls for create database (#713) 2023-10-06 13:57:18 +00:00
e63b9bdcd4 add tests (#705)
* test fork namespace

* test destroy namespace

* test create namespace load from url

test load from dump

* test load dump from file

* reorganize test dir

* fix sync_many_replicas test

* load dump tests

* test malformed dumps

* add create dump test.

* fix tests
2023-10-06 13:46:28 +00:00
0c854de9a6 bottomless-cli: add verify command (#744)
* bottomless-cli: fix too eager create_dir_all() call

The way create_dir_all() is called on a path to the database,
it also creates the final file name as a directory. As a result,
restoration later fails, because `data.tmp` cannot be moved to `data`,
with `data` being already mistakenly created as a directory.

* bottomless-cli: add `verify` command

The command works similarly to `restore`, except it restores
to a temporary directory and runs a `pragma integrity_check` query.

On success it should return:
```
bottomless-cli -e http://localhost:9000 -n ns-:default -d e4e57664-01ad-76cc-9f19-a96700d5b2e4 verify
Snapshot size: 13512704
Verification: ok
```
2023-10-06 14:09:52 +02:00
d606792779 remove useless patches (#742) 2023-10-06 10:27:01 +00:00
a5a8ea2425 libsql: fix a typo around vacuum limits (#743)
It says 128MiB, but it was later changed to 32MiB, and the comment
was not updated properly.
2023-10-06 10:25:35 +00:00
02dc976ae2 Remove page_size recommendation in bottomless (#741) 2023-10-06 09:40:48 +00:00
affa973fd9 add commit sha suffix to nightly build (#740) 2023-10-06 09:30:57 +00:00
d79f85c2d9 Vacuum before checkpointing (#739)
* bottomless: remove false positive [BUG] message

For a transaction that spans multiple xFrames calls, it's ok
if our last_valid_frame doesn't match the one that came with xFrames.
As long as our last_valid_frame is *not greater* than the one reported,
we're good. If it's greater, we're in trouble and we report a bug.

* connection: vacuum before checkpointing

This commit adds an optional VACUUM operation before our
periodic checkpoint -- since VACUUM should always be followe
by a checkpoint.
The are two criteria to qualify for vacuuming:
1. We have at least 32MiB of data
2. There are more free pages in the db than regular ones

If that's the case, we keep the db file more than 2x larger
than it could be, and that calls for a vacuum.
Also, for small databases, we entirely skip vacuuming.

Fixes #734
2023-10-05 13:00:10 +00:00
298b8649ce Limit maximum DB size (#736)
* add max_db_size config option

* limit db size on connection creation

* fmt

* allow setting max db size on namespace create
2023-10-05 10:10:41 +00:00
f8c3d94da9 hrana: add diagnostics for connections (#729)
* hrana: add diagnostics for connections

This commit adds a /v2/diagnostics endpoint which prints
various information about current hrana-over-http connections.

Draft, because the diagnostics are currently in a very debuggy
format, and I'm figuring out if we can make it more human-readable.
Still, they're enough to determine if something is holding a lock
via an abandoned hrana-over-http stream.

Example:
```
$ curl -s http://localhost:8080/v2/diagnostics | jq
[
  "expired",
  "expired",
  "expired",
  "expired",
  "expired",
  "(conn: Mutex { data: <locked> }, timeout_ms: 872, stolen: false)",
  "(conn: Mutex { data: <locked> }, timeout_ms: 0, stolen: true)"
]
```

* apply review suggestions: no more Debug required in WalHook

* apply review fixes: move everything to admin api

* Update sqld/src/http/admin/mod.rs

Co-authored-by: ad hoc <postma.marin@protonmail.com>

* fix Json return type

* revert leftover change - returning from passive checkpoints

---------

Co-authored-by: ad hoc <postma.marin@protonmail.com>
2023-10-05 11:30:17 +02:00
1713ebab47 Fix typo of "incoming" tracing info message (#737)
* Fix typo of "incoming" tracing info message

* Remove git submodules instructions from docs

Git submodules is no longer used in this repo
2023-10-05 04:54:33 +00:00
81dafadf38 bump sqld to v0.21.7 (#735) 2023-10-04 17:31:30 +00:00
2bcc699ced bottomless restore to temp file first, replace only on success (#732)
* bottomless restore to temp file first, replace only on success

* fixed clippy errors
2023-10-04 14:44:35 +00:00
a0984d34c1 don't snapshot empty db files (#730) 2023-10-04 13:42:12 +00:00
15f8895e45 bump sqld to v0.21.6 (#728) 2023-10-04 07:52:30 +00:00
db70429118 changed the path where bottomless is storing gzipped db snapshot (#727)
* changed the path where bottomless is storing gzipped db snapshot

* fixed clippy suggestions
2023-10-04 06:47:14 +00:00
a66d3b3f95 bottomless: checkpoint before initializing bottomless (#726)
* bottomless: checkpoint before initializing bottomless

Due to a bug in wallog recovery, we need to checkpoint
the database *strictly before* we initialize bottomless.
A proper fix should be to use our virtual WAL methods
for checkpointing, but there's an initialization cycle
and resolving it will be a larger patch - a connection
with WAL methods wants us to already have the replication
logger created, and replication logger wants to perform
a checkpoint on creation.
As a mid-term solution, we just perform the forbidden
regular checkpoint before bottomless is ever launched.
Combined with the fact that bottomless treats existing
databases as the source of truth, it just creates
a new backup generation and continues working properly.

The following scenario was buggy before:
 1. We leave the db in the state where some WAL frames
   still exist in data-wal file
 2. We restart sqld
 3. bottomless is initialized, it reuses the existing db
    and WAL frames and uploads them to S3, to avoid
    creating a potentially costly snapshot
 4. ReplicationLogger::new() incorrectly calls
    sqlite3_wal_checkpoint which swipes data from under
    bottomless.
 5. Bottomless thinks it hasn't checkpointed and continues
    to write WAL frames. As a result, it writes garbage
    to S3, because the db was checkpointed outside
    of bottomless control

* fmt fix
2023-10-03 15:02:28 +00:00