0
0
mirror of https://github.com/tursodatabase/libsql.git synced 2025-04-12 01:30:02 +00:00
Commit Graph

59 Commits

Author SHA1 Message Date
bd3e0d6904 Merge pull request from tursodatabase/vector-search-fix-integrity-check
vector search: fix integrity check
2024-08-23 11:21:53 +00:00
2f15a0e060 Merge pull request from tursodatabase/hide-vector-index-for-sqlite-planner
vector search: hide vector index for sqlite planner
2024-08-23 08:45:12 +00:00
0d8ca0c715 rename BFLOAT16 to FLOATB16 everywhere 2024-08-22 12:59:26 +04:00
74346590f3 implement and integrate bfloat16 2024-08-22 12:32:49 +04:00
c5dbc8c4a1 fix integrity check for vector indices 2024-08-22 12:15:03 +04:00
69bce662bf properly integrate float16 2024-08-22 00:48:30 +04:00
6e67b36c32 add COUNT(*) test with partial index 2024-08-19 20:16:12 +04:00
f2271b04a8 Merge pull request from tursodatabase/vector-search-int8
vector search: int8 support
2024-08-16 09:11:04 +00:00
fd1f5c4dee Merge pull request from tursodatabase/vector-search-full-support-all-column-types
vector search: full support all column types
2024-08-14 16:35:56 +00:00
bd25344573 add tests and refine error messages 2024-08-13 18:08:08 +04:00
34c297451e support float8 neighbors compression 2024-08-13 17:41:11 +04:00
244a34a16f fix bug in cosine distance 2024-08-13 17:21:38 +04:00
8a2a019b1b add implementation of float8 vector type (int8 quantization) 2024-08-13 16:22:40 +04:00
7cafb80b46 use float1bit instead of 1bit everywhere in the code in index settings 2024-08-13 11:24:46 +04:00
30a198e962 refine error messages 2024-08-12 17:36:02 +04:00
e0035d6e17 add more tests 2024-08-12 17:25:29 +04:00
1a704d8ba4 fix test 2024-08-12 17:19:40 +04:00
307139fda9 add conversion tests 2024-08-12 16:15:24 +04:00
4a57f02b78 add simple conversion test 2024-08-12 15:20:52 +04:00
0f4531a10e add support for vector indices over f64 embeddings 2024-08-12 13:45:38 +04:00
c135391f60 fix unstable test 2024-08-12 12:16:52 +04:00
cea87253fb fix DELETE from vector index as there can be no row due to the NULL value of the vector 2024-08-12 11:43:51 +04:00
90007425ba add simple pragma test 2024-08-11 19:08:38 +04:00
8262d238f5 Merge pull request from tursodatabase/vector-search-compression
vector search: neighbors compression (1bit quantization)
2024-08-11 09:41:58 +00:00
8441108e71 allow vector index to be partial 2024-08-08 18:31:38 +04:00
11e6a93264 add simple test 2024-08-08 14:56:42 +04:00
f80444ac45 fix test 2024-08-08 14:50:43 +04:00
76558e8df4 fix behaviour of VACUUM for vector indices to make rowid consistent
between shadow tables and base table
2024-08-06 13:33:23 +04:00
c03c00b77e Merge pull request from penberg/merge-sqlite-3.45.1-new
Merge upstream SQLite 3.45.1
2024-08-01 09:09:15 +00:00
90dbcff1df improve vector index key handling code 2024-07-26 22:51:36 +04:00
9981d607a7 refine errors even more 2024-07-26 18:54:58 +04:00
ad766c429a refine errors for vector search 2024-07-26 18:38:50 +04:00
675cdd4710 rename max_edges to max_neighbors 2024-07-25 19:26:40 +04:00
ea1bb70e80 add simple test which mention all index parameters 2024-07-25 19:26:40 +04:00
9057bcb6cb add more tests for vector feature 2024-07-25 15:49:09 +04:00
9ed72eb5ae Merge upstream SQLite 3.45.1
This merges the version-3.45.1 tag from upstream SQLite git repository
to libSQL with the following conflicts resolved:

Conflicts:
      README.md
      ext/jni/src/org/sqlite/jni/capi/ConfigSqllogCallback.java
      libsql-sqlite3/configure
      libsql-sqlite3/doc/jsonb.md
      libsql-sqlite3/ext/fts5/test/fts5faultH.test
      libsql-sqlite3/ext/fts5/test/fts5origintext.test
      libsql-sqlite3/ext/fts5/test/fts5origintext2.test
      libsql-sqlite3/ext/fts5/test/fts5origintext3.test
      libsql-sqlite3/ext/fts5/test/fts5origintext4.test
      libsql-sqlite3/ext/fts5/test/fts5origintext5.test
      libsql-sqlite3/ext/fts5/test/fts5secure8.test
      libsql-sqlite3/ext/fts5/test/fts5tokenizer2.test
      libsql-sqlite3/ext/fts5/test/fts5trigram2.test
      libsql-sqlite3/ext/jni/src/org/sqlite/jni/annotation/Experimental.java
      libsql-sqlite3/ext/jni/src/org/sqlite/jni/capi/ConfigSqlLogCallback.java
      libsql-sqlite3/ext/jni/src/org/sqlite/jni/capi/ConfigSqllogCallback.java
      libsql-sqlite3/ext/jni/src/org/sqlite/jni/wrapper1/WindowFunction.java
      libsql-sqlite3/ext/wasm/GNUmakefile
      libsql-sqlite3/ext/wasm/batch-runner-sahpool.html
      libsql-sqlite3/ext/wasm/batch-runner-sahpool.js
      libsql-sqlite3/src/pager.c
      libsql-sqlite3/src/shell.c.in
      libsql-sqlite3/src/sqliteInt.h
      libsql-sqlite3/src/wal.c
      libsql-sqlite3/test/fts3integrity.test
      libsql-sqlite3/test/json/jsonb-q1.txt
      libsql-sqlite3/test/json106.test
      libsql-sqlite3/test/json107.test
      libsql-sqlite3/test/jsonb01.test
      libsql-sqlite3/test/mmapcorrupt.test
      libsql-sqlite3/test/releasetest_data.tcl
      libsql-sqlite3/test/shell9.test
      libsql-sqlite3/test/wapp.tcl
      libsql-sqlite3/test/wapptest.tcl
2024-07-25 13:45:06 +03:00
ccc9f58d51 fix dump and vacuum of vector indices ()
* fix dump and vacuum of vector indices

* fix comment a bit

* build bundles

* slightly improve test

* review fixes

* build bundles
2024-07-25 07:26:41 +00:00
5e6afb386d Vector search respect db idx ()
* propagate schema name (iDb) for vector index to support working not only with main DB

* add basic test

* sometimes zDbSName can be null and this is fine

* avoid test from writing files to disk

* build bundles
2024-07-22 18:29:11 +00:00
37fc1a1d88 integrate diskann to the sqlite code ()
* integrate diskann to the sqlite code

* cleanup code a bit and add more comments

* make parsing code resilient to spaces inside FLOAT typename

* fixup

* fix bugs related to deletes in diskann

* small cleanup

* rename macro

* slightly cleanup vtab code

* slightly improve vectorIndex code

* make code less hacky

* add strange test

* build bundles

* return unit test for diskann pieces

* add one more test

* don't run search on first insertion

* review fixes

* disable index creation in non-normal parse modes

* build bundles
2024-07-19 16:57:11 +00:00
be1fca578f feat(vector): add basic functions (vector/vector_extract etc) ()
* add basic functions for working with f32/f64 vectors

* add header file

* fix comment style

* add generic code for working with vectors

* register vector functions

* fix edge cases in vector extract impl

* add static flag for vector (will be used later)

* add basic TCL-based tests

* guard vector functions with SQLITE_OMIT_VECTOR

* adjust build scripts

* delete libsql_vector_idx from this branch

* add one more test

* update bundles

* fix asserts

* fix constants (#define-s are universal but const int doesn't recognized by all compilers)

* update bundles

* review fixes

* build bundles

* add comment about linking math library
2024-07-08 08:27:33 +00:00
2402249b5d apply alter column command only to the table ()
* add alter column test

* apply alter column only to the table and ignore indices/triggers/views etc

* fix tests

* update bundle

* add test in rust_suite
2024-07-04 19:48:33 +00:00
b17b6ee0d4 adjust ALTER COLUMN parsing ()
* add basic TCL tests for ALTER COLUMN libsql feature

* propagate length of the new column definition directly from the parser

- this will allow libsql to automatically handle comment and space characters appended to the column definition

* small formatting fixes

* cargo xtask build-bundled

* add test against sqlite3 source in rust_suite

* fix test in rust_suite
2024-07-02 22:02:14 +00:00
e451079c18 Allow single transaction in schema migration ()
* Return also Program from Connection::run

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Extract LibSqlConnection::execute

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Use LibSqlConnection::execute in write_proxy

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Remove Clone from Program

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Remove Arc from Program

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* Allow a single transaction in schema migration

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

* fix rustc-hash dep issue

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>

---------

Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
2024-05-28 15:05:45 +00:00
ef446125f7 write queue ()
* introduce connection manager

* remove unused wal methods

* remove lock stealer

* Make use of ConnectionManager in LibsqlConnection

it now takes a W: WalWrap instead of a WalManager. This is because we
want to inject the connection manager at the bottom of the wal wrapping
chain.

* add missing deps

* turn ReplicationLogger into a WrapWal

* update spots to to pass wal wrapper instead of wal manager

* remove dbg

* fmt

* fix sqlite3 rust tests
2024-03-20 22:27:13 +00:00
112c72bb8e bugfix: RANDOM ROWID should be respected in VACUUM and bulk insert ops ()
* Add regression tests for random rowid bugs

Tests added for:

1. VACUUM does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/929
2. Bulk insert into a fresh table does not respect RANDOM ROWID - https://github.com/tursodatabase/libsql/issues/1046

In both cases, the table should end up random rowids, but they seem to have sequential ones.

* bugfix: pass `regNextRowid` to `OP_NewRowid` call in `xferOptimization`

The register `regNextRowid` contains the `LIBSQL_RANDOM_ROWID_MARKER` value
which tells the VDBE to use a random value for row ids. The method `xferOptimization`
is used in `VACUUM` and bulk insert routines where data from one table is
inserted in another. Since the data is inserted in a loop, it checks for
max rowid once and then keeps inserting it serially. Hence we pass the marker
in each call, so that random id is generated.
2024-02-19 16:58:50 +00:00
d178de8f07 bugfix: reset modifications of sqlite3_vtab ()
* Remove `libsql_module` field from `sqlite3_vtab`

* remove `libsql_module` struct

Also removed the associated functions `libsql_create_module_v2`, `libsql_create_module` functions'.

The `libsql_module` had a function `xPreparedSql` which is now moved to `sqlite_module`. The `sqlite_module` might get changed in the upstream, so added some padding space for our custom functions

* generate ffi bindings

* Add a github workflow to test crsqlite

* fix crsqlite: remove `pLibsqlModule` references

* Add tests for sqlite-vss extension
2024-02-15 18:18:30 +00:00
b0887d6fa6 fix ci () 2024-02-07 08:54:01 +00:00
a72c066a8e Wal related changes ()
wal extensions
2024-02-06 17:44:23 +00:00
e1cc9ff829 return number of committed frames on insert_frames () 2024-01-25 16:39:29 +00:00
62f61dbf14 review edits 2024-01-19 12:16:25 +01:00