0
0
mirror of https://github.com/tursodatabase/libsql.git synced 2025-05-20 11:28:10 +00:00
Commit Graph

46 Commits

Author SHA1 Message Date
915ab1f4ba configure yystackdepth 2025-03-22 14:00:36 +04:00
3ad67a8ecf fix: only write to OUT_DIR 2025-03-17 09:11:08 -03:00
da54c1bf97 fix: try fixing copy, again 2025-03-17 09:11:08 -03:00
5e20bd300c fix: use copy_with_cp everywhere 2025-03-17 09:11:08 -03:00
7a941c6746 fix: add --no-preserve for nix on macos 2025-03-17 09:11:08 -03:00
8408283b37 feat: add support for windows cross-compilation 2025-03-17 09:11:08 -03:00
c74c8eef99 refactor+fix: trying to fix sqlite3mc cross compilation 2025-03-17 09:11:08 -03:00
d09ed8230d fix: cross compilation from aarch64 to x86_64 MacOS 2025-03-17 09:11:08 -03:00
e14762daf5 add libsql-disable-checkpoint-downgrade cargo feature in libsql-ffi crate 2025-02-27 13:08:34 +04:00
a8c11db984 fix: add sqlite3 includes 2025-01-28 13:01:16 -03:00
d3a156caf5 bundle SQLean extensions
A common complain with libSQL is how to run extensions. The main
mechanism, with a .so, has a lot of issues around how those .so are
distributed.

The most common extensions are the ones in the sqlean package. We can
improve this experience by bundling them in our sqlite build.

Not all SQLean extensions are kosher: some of them, like fileio, use
the vfs. Others, are deemed too complex.

The extensions included here are a subset that we deem important enough,
and low risk enough, to just be a part of the main bundle.
2025-01-16 22:25:16 -05:00
13385969d1 libsql-ffi: Fix build on macOS
The `cp` command does not support `--no-preserve` on macOS.
2024-12-04 17:44:43 +02:00
38cfab7d5d Revert "Revert "Merge pull request from awakecoding/libsql-ffi-build-cp-fix""
This reverts commit 9499c3c40a. The real
problem is not Windows, but build on macOS. Let's revert this revert and
fix macOS instead.
2024-12-04 17:44:22 +02:00
9499c3c40a Revert "Merge pull request from awakecoding/libsql-ffi-build-cp-fix"
This reverts commit 5ee4723699, reversing
changes made to cc1d9c5afd because it breaks `libsql-js` build on macOS:

```
penberg@vonneumann libsql-js % cargo check
    Updating git repository `https://github.com/tursodatabase/libsql/`
   Compiling libsql-ffi v0.5.0 (https://github.com/tursodatabase/libsql/?rev=9c73c68f0b034c5e01a51d9dc93a61936a71c801#9c73c68f)
   Compiling libsql-sqlite3-parser v0.13.0 (https://github.com/tursodatabase/libsql/?rev=9c73c68f0b034c5e01a51d9dc93a61936a71c801#9c73c68f)
    Checking reqwest v0.12.9
    Checking libsql-hrana v0.2.0 (https://github.com/tursodatabase/libsql/?rev=9c73c68f0b034c5e01a51d9dc93a61936a71c801#9c73c68f)
error: failed to run custom build command for `libsql-ffi v0.5.0 (https://github.com/tursodatabase/libsql/?rev=9c73c68f0b034c5e01a51d9dc93a61936a71c801#9c73c68f)`

Caused by:
  process didn't exit successfully: `/Users/penberg/src/tursodatabase/libsql-js/target/debug/build/libsql-ffi-755a07789947d931/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-link-lib=framework=Security
  cargo:rerun-if-changed=bundled/src/sqlite3.c
  cargo:rerun-if-changed=/Users/penberg/src/tursodatabase/libsql-js/target/debug/build/libsql-ffi-e660048e9fb948a4/out/sqlite3mc/libsqlite3mc_static.a

  --- stderr
  cp: illegal option -- -
  usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file target_file
         cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file ... target_directory
  thread 'main' panicked at /Users/penberg/.cargo/git/checkouts/libsql-311658d335deb3b1/9c73c68/libsql-ffi/build.rs:43:44:
  called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "Failed to copy using cp" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
```
2024-11-15 10:55:30 +02:00
6a7a3e50a1 fix libsql-ffi build.rs usage of cp command on Windows 2024-10-22 16:15:29 -04:00
e83b13b1d2 libsql: fix musl builds failing due to fcntl64
This fixes a build issue with musl based systems that do not have
fcntl64 in their libc. This is done by setting `FILE_OFFSET_BITS=32`
only for musl targets. This forces the compiler to not convert `fcntl`
to `fcntl64`.
2024-09-09 13:40:44 -04:00
e349a06961 Make libsql-ffi's build more friendly to sandboxed environments. ()
fix: prevent out_path from inheriting the mode of source bindgen.rs
2024-06-18 12:55:36 +00:00
cb89097a57 ffi, build: Don't use UNC paths on Windows ()
Windows build fails with "The network name cannot be found" error when
built with encryption. That's because the std::fs::canocalize() function
returns UNC paths on Windows, which confuses CMake to fail.

Switch to std::env::current_dir() instead, which returns paths that
CMake can deal with.
2024-06-07 18:02:29 +00:00
17a687b0bb Fix cross build ()
* Revert "libsql-ffi: Make CC and CXX configurable ()"

This reverts commit 897ae8beac, because
it's bogus and completely ignores the environment variables in a cross
environment.

* ffi: Fix cross builds

Let's use the `TARGET` environment variable to determine what the
build target is to fix cross build falures in libsql-js.git:

```
error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/rust/bin" VSLANG="1033" "aarch64-linux-gnu-gcc" "-Wl,--version-script=/tmp/rustcualN23/list" "-Wl,--no-undefined-version" "/tmp/rustcualN23/symbols.o" "/target/aarch64-unknown-linux-gnu/release/deps/libsql_js.libsql_js.daf3718404b8afc6-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-gnu/release/deps" "-L" "/target/release/deps" "-L" "/target/aarch64-unknown-linux-gnu/release/build/ring-87b9427ec69caec7/out" "-L" "/target/aarch64-unknown-linux-gnu/release/build/libsql-ffi-f019579845159c8e/out" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/rustcualN23/libring-d3d989c5f3051615.rlib" "/tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-6043da066acc1466.rlib" "-Wl,-Bdynamic" "-ldl" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/target/aarch64-unknown-linux-gnu/release/deps/liblibsql_js.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
          /tmp/rustcualN23/liblibsql_ffi-54e0e58458c35432.rlib: error adding symbols: File in wrong format
          collect2: error: ld returned 1 exit status
```
2024-06-07 16:39:39 +00:00
5129bdb2a3 libsq wal integration ()
* integrate libsql-wal

* fix wal bug

* shutdown registry on shutdown

* fmt
2024-06-03 09:30:40 +00:00
897ae8beac libsql-ffi: Make CC and CXX configurable ()
Make CC and CXX configurable for the build process. Allows us to, for
example, make sure we're building with just clang on macOS.
2024-04-02 14:40:30 +00:00
c45e72d0a6 libsql-ffi: Fix sqlite3mc build output directory ()
Don't build inside the source tree because `cargo clean` won't clean up
after it...
2024-03-25 16:45:26 +00:00
234e5d4871 Fixed function make_amalgamation naming typo ()
Use `amalgamation` instead of `amalgation`.

Ref: https://www.merriam-webster.com/dictionary/amalgamation
2024-03-12 10:27:42 +00:00
47fef88f19 ffi: blocklist wasm fns from bindgen () 2024-03-07 20:13:39 +00:00
fd6afb784f libsql-ffi: Disable stack protector for sqlite3mc on musl ()
The stack protector links to symbols like `strcpy_chk`, which are not
available on musl. Therefore, disable it.

Fixes 
2024-03-07 11:59:26 +00:00
0cb88082ef Fix build ()
Signed-off-by: Piotr Jastrzebski <piotr@chiselstrike.com>
2024-02-28 14:08:03 +00:00
8fd841d87d libsq-ffi: Drop git submodule trickery from sqlite3mc build ()
As Lucio points out, we no longer do submodules for sqlite3mc so let's
just drop the bogus `git submodule update` invocation.
2024-02-16 15:31:52 +00:00
86172b548e Fix sqlite3mc build ()
* libsql-ffi: Enable PIC for sqlite3mc build

I am seeing the following build error when building the libsql-ffi crate
on Linux/x86-64:

```
/usr/bin/ld:
/tmp/rustcYVhHYo/liblibsql_ffi-20adf4ca11d8e922.rlib(sqlite3mc.c.o):
relocation R_X86_64_32S against `.rodata' can not be used when making a
shared object; recompile with -fPIC
```

Let's turn on PIC for sqlite3mc to fix the problem.

* libsql-ffi: Fix sqlite3mc build under `cross`

We need to use the C compiler configured by `cross` with sqlite3mc;
otherwise we end up building with the host compiler, resulting in errors
like this:

```
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /tmp/rustcJ70WRo/liblibsql_ffi-5d7e6e8037dbda37.rlib(sqlite3mc.c.o): Relocations in generic ELF (EM: 62)
```

* libsql-ffi: Rewrite build_libsqlite3mc.sh in Rust

The bash shell script is turning out to be painful to run on Windows so
let's just rewrite the logic in Rust and place it in `build.rs`.
2024-02-14 15:26:16 +00:00
02b9e82945 chore: don't rebuild ffi without ciphers () 2024-02-13 16:31:11 +00:00
37fb629f50 run clippy () 2024-01-31 08:42:20 +00:00
f05f349c06 chore: fix ffi build with all features ()
* chore: fix ffi build with all features

* set flags only for clang

* Update libsql-ffi/build.rs
2024-01-25 21:03:11 +00:00
58eda02bb3 chore: fix ffi build script again ()
* chore: fix ffi build script again

* ffi: separate building and copying for cipher code

* build dylib if it doesn't exist

* fix wasm builds with ciphers
2024-01-24 22:02:51 +00:00
975646e034 libsql: add local sync write delegation ()
* libsql: add local sync write delegation

* add additional query check to local sync test
2024-01-24 20:39:04 +00:00
dd0e23c81c fix docker build 2024-01-24 11:03:03 +01:00
d9a9f44ecd libsql-ffi: add c11 flag to make build compatible with old compilers 2024-01-22 21:43:07 -03:00
ad5ff8d171 libsql-ffi: deduplicate sqlite3.c amalgamations
From now on, SQLite3MultipleCiphers will use the amalgamation
file kept in libsql-ffi/bundled/src.
2024-01-19 08:22:57 +01:00
d149ecebd8 libsql-ffi: compile-in Wasm conditionally with encryption enabled 2024-01-19 08:22:57 +01:00
dedbdd03b3 Merge pull request from tursodatabase/sqlite3mc_part0
Encryption at rest preparations: integrate with SQLite3MultipleCiphers
2024-01-11 18:23:01 +00:00
f0929037d1 chore: fix ffi build script 2024-01-10 09:53:19 -05:00
c504f8311b libsql-ffi: move multiple-ciphers support to a feature
Enabled with `cargo build -F multiple-ciphers`
2024-01-09 18:12:02 +01:00
c74fb49f5c still hacky, but compiles lib instead of shipping precompiled 2024-01-09 18:12:02 +01:00
253a295865 encryption demo: based on precompiled libsqlite3mc.so
** DEEP, DEEP DRAFT **

The prebuilt library is from my fork:
https://github.com/psarna/SQLite3MultipleCiphers

The key is hardcoded to "heyhey".

After you run sqld with this patch, all data is encoded on disk
with a "heyhey" key. You can't read it directly from the file,
unless you use sqlite3mc's shell and start with
 > PRAGMA KEY=heyhey;

, and then it gets properly decrypted.

** TODO **
1. We need to adopt SQLite3MultipleCiphers source code and integrate
   with our build system, if we want to use it.
2. Pretty sure the hardcoded "heyhey" passphrase won't pass SOC2,
   but I need to consult that with a lawyer.
2024-01-09 18:12:02 +01:00
30f998cac9 sqld: small debug additions and ignore clean up 2023-12-11 15:39:49 -05:00
425b370fb1 fix ffi build.rs 2023-12-08 17:57:34 +01:00
176d6c4376 add wasmtime-bindings feature to libsql-sys 2023-11-30 11:24:51 +01:00
477af9bef6 introduce libsql-ffi 2023-11-15 11:08:51 +01:00