2022-08-10 11:26:08 +00:00
/ *
2022 - 07 - 22
The author disclaims copyright to this source code . In place of a
legal notice , here is a blessing :
* May you do good and not evil .
* May you find forgiveness for yourself and forgive others .
* May you share freely , never taking more than you give .
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2022-08-17 16:44:05 +00:00
This file implements the initializer for the sqlite3 " Worker API
# 1 " , a very basic DB access API intended to be scripted from a main
window thread via Worker - style messages . Because of limitations in
that type of communication , this API is minimalistic and only
capable of serving relatively basic DB requests ( e . g . it cannot
process nested query loops concurrently ) .
This file requires that the core C - style sqlite3 API and OO API # 1
have been loaded .
* /
/ * *
2022-08-25 11:39:12 +00:00
sqlite3 . initWorker1API ( ) implements a Worker - based wrapper around
SQLite3 OO API # 1 , colloquially known as "Worker API #1" .
2022-08-10 11:26:08 +00:00
In order to permit this API to be loaded in worker threads without
automatically registering onmessage handlers , initializing the
2022-08-25 11:39:12 +00:00
worker API requires calling initWorker1API ( ) . If this function is
called from a non - worker thread then it throws an exception . It
must only be called once per Worker .
2022-08-10 11:26:08 +00:00
2022-08-17 16:44:05 +00:00
When initialized , it installs message listeners to receive Worker
messages and then it posts a message in the form :
2022-08-10 11:26:08 +00:00
` ` `
2022-08-25 11:39:12 +00:00
{ type : 'sqlite3-api' , result : 'worker1-ready' }
2022-08-10 11:26:08 +00:00
` ` `
2022-08-17 16:44:05 +00:00
to let the client know that it has been initialized . Clients may
optionally depend on this function not returning until
initialization is complete , as the initialization is synchronous .
In some contexts , however , listening for the above message is
a better fit .
2022-08-24 18:39:46 +00:00
Note that the worker - based interface can be slightly quirky because
of its async nature . In particular , any number of messages may be posted
to the worker before it starts handling any of them . If , e . g . , an
"open" operation fails , any subsequent messages will fail . The
Promise - based wrapper for this API ( ` sqlite3-worker1-promiser.js ` )
is more comfortable to use in that regard .
2022-08-25 11:39:12 +00:00
The documentation for the input and output worker messages for
this API follows ...
2022-08-24 18:39:46 +00:00
2022-08-25 11:39:12 +00:00
=== === === === === === === === === === === === === === === === === === === === === === ==
Common message format ...
Each message posted to the worker has an operation - independent
envelope and operation - dependent arguments :
` ` `
{
type : string , // one of: 'open', 'close', 'exec', 'config-get'
messageId : OPTIONAL arbitrary value . The worker will copy it as - is
into response messages to assist in client - side dispatching .
dbId : a db identifier string ( returned by 'open' ) which tells the
operation which database instance to work on . If not provided , the
first - opened db is used . This is an "opaque" value , with no
inherently useful syntax or information . Its value is subject to
change with any given build of this API and cannot be used as a
basis for anything useful beyond its one intended purpose .
args : ... operation - dependent arguments ...
// the framework may add other properties for testing or debugging
// purposes.
}
` ` `
Response messages , posted back to the main thread , look like :
` ` `
{
type : string . Same as above except for error responses , which have the type
'error' ,
messageId : same value , if any , provided by the inbound message
dbId : the id of the db which was operated on , if any , as returned
by the corresponding 'open' operation .
result : ... operation - dependent result ...
}
` ` `
=== === === === === === === === === === === === === === === === === === === === === === ==
Error responses
Errors are reported messages in an operation - independent format :
` ` `
{
type : 'error' ,
messageId : ... as above ... ,
dbId : ... as above ...
result : {
operation : type of the triggering operation : 'open' , 'close' , ...
message : ... error message text ...
errorClass : string . The ErrorClass . name property from the thrown exception .
input : the message object which triggered the error .
stack : _if available _ , a stack trace array .
}
}
` ` `
=== === === === === === === === === === === === === === === === === === === === === === ==
"config-get"
This operation fetches the serializable parts of the sqlite3 API
configuration .
Message format :
` ` `
{
type : "config-get" ,
messageId : ... as above ... ,
args : currently ignored and may be elided .
}
` ` `
Response :
` ` `
{
type : 'config' ,
messageId : ... as above ... ,
result : {
persistentDirName : path prefix , if any , of persistent storage .
An empty string denotes that no persistent storage is available .
bigIntEnabled : bool . True if BigInt support is enabled .
persistenceEnabled : true if persistent storage is enabled in the
current environment . Only files stored under persistentDirName
will persist , however .
}
}
` ` `
=== === === === === === === === === === === === === === === === === === === === === === ==
"open" a database
Message format :
` ` `
{
type : "open" ,
messageId : ... as above ... ,
args : {
filename [ = ":memory:" or "" ( unspecified ) ] : the db filename .
See the sqlite3 . oo1 . DB constructor for peculiarities and transformations ,
persistent [ = false ] : if true and filename is not one of ( "" ,
":memory:" ) , prepend sqlite3 . capi . sqlite3 _web _persistent _dir ( )
to the given filename so that it is stored in persistent storage
_if _ the environment supports it . If persistent storage is not
supported , the filename is used as - is .
}
}
` ` `
Response :
` ` `
{
type : 'open' ,
messageId : ... as above ... ,
result : {
filename : db filename , possibly differing from the input .
dbId : an opaque ID value which must be passed in the message
envelope to other calls in this API to tell them which db to
use . If it is not provided to future calls , they will default to
operating on the first - opened db . This property is , for API
consistency ' s sake , also part of the contaning message envelope .
Only the ` open ` operation includes it in the ` result ` property .
persistent : true if the given filename resides in the
known - persistent storage , else false . This determination is
independent of the ` persistent ` input argument .
}
}
` ` `
=== === === === === === === === === === === === === === === === === === === === === === ==
"close" a database
Message format :
` ` `
{
type : "close" ,
messageId : ... as above ...
dbId : ... as above ...
args : OPTIONAL : {
unlink : if truthy , the associated db will be unlinked ( removed )
from the virtual filesystems . Failure to unlink is silently
ignored .
}
}
` ` `
If the dbId does not refer to an opened ID , this is a no - op . The
inability to close a db ( because it ' s not opened ) or delete its
file does not trigger an error .
Response :
` ` `
{
type : 'close' ,
messageId : ... as above ... ,
result : {
filename : filename of closed db , or undefined if no db was closed
}
}
` ` `
=== === === === === === === === === === === === === === === === === === === === === === ==
"exec" SQL
All SQL execution is processed through the exec operation . It offers
most of the features of the oo1 . DB . exec ( ) method , with a few limitations
imposed by the state having to cross thread boundaries .
Message format :
` ` `
{
type : "exec" ,
messageId : ... as above ...
dbId : ... as above ...
args : string ( SQL ) or { ... see below ... }
}
` ` `
Response :
` ` `
{
type : 'exec' ,
messageId : ... as above ... ,
dbId : ... as above ...
result : {
input arguments , possibly modified . See below .
}
}
` ` `
The arguments are in the same form accepted by oo1 . DB . exec ( ) , with
the exceptions noted below .
A function - type args . callback property cannot cross
the window / Worker boundary , so is not useful here . If
args . callback is a string then it is assumed to be a
message type key , in which case a callback function will be
applied which posts each row result via :
postMessage ( { type : thatKeyType ,
rowNumber : 1 - based - # ,
row : theRow ,
columnNames : anArray
} )
And , at the end of the result set ( whether or not any result rows
were produced ) , it will post an identical message with
( row = undefined , rowNumber = null ) to alert the caller than the result
set is completed . Note that a row value of ` null ` is a legal row
result for certain arg . rowMode values .
( Design note : we don ' t use ( row = undefined , rowNumber = undefined ) to
indicate end - of - results because fetching those would be
indistinguishable from fetching from an empty object unless the
client used hasOwnProperty ( ) ( or similar ) to distinguish " missing
property " from " property with the undefined value " . Similarly ,
` null ` is a legal value for ` row ` in some case , whereas the db
layer won ' t emit a result value of ` undefined ` . )
The callback proxy must not recurse into this interface . An exec ( )
call will type up the Worker thread , causing any recursion attempt
to wait until the first exec ( ) is completed .
The response is the input options object ( or a synthesized one if
passed only a string ) , noting that options . resultRows and
options . columnNames may be populated by the call to db . exec ( ) .
2022-08-24 18:39:46 +00:00
2022-08-10 11:26:08 +00:00
* /
2022-08-22 13:34:13 +00:00
self . sqlite3ApiBootstrap . initializers . push ( function ( sqlite3 ) {
sqlite3 . initWorker1API = function ( ) {
2022-08-10 11:26:08 +00:00
'use strict' ;
const toss = ( ... args ) => { throw new Error ( args . join ( ' ' ) ) } ;
if ( 'function' !== typeof importScripts ) {
toss ( "Cannot initalize the sqlite3 worker API in the main thread." ) ;
}
const self = this . self ;
2022-08-11 15:45:32 +00:00
const sqlite3 = this . sqlite3 || toss ( "Missing this.sqlite3 object." ) ;
const SQLite3 = sqlite3 . oo1 || toss ( "Missing this.sqlite3.oo1 OO API." ) ;
2022-08-10 11:26:08 +00:00
const DB = SQLite3 . DB ;
/ * *
Returns the app - wide unique ID for the given db , creating one if
needed .
* /
const getDbId = function ( db ) {
let id = wState . idMap . get ( db ) ;
if ( id ) return id ;
id = 'db#' + ( ++ wState . idSeq ) + '@' + db . pointer ;
/ * * ^ ^ ^ c a n ' t s i m p l y u s e d b . p o i n t e r b / c c l o s i n g / o p e n i n g m a y r e - u s e
the same address , which could map pending messages to a wrong
instance . * /
wState . idMap . set ( db , id ) ;
return id ;
} ;
/ * *
2022-08-24 18:39:46 +00:00
Internal helper for managing Worker - level state .
2022-08-10 11:26:08 +00:00
* /
const wState = {
2022-08-25 11:39:12 +00:00
/ * * F i r s t - o p e n e d d b i s t h e d e f a u l t f o r f u t u r e o p e r a t i o n s w h e n n o
dbId is provided by the client . * /
2022-08-10 11:26:08 +00:00
defaultDb : undefined ,
2022-08-25 11:39:12 +00:00
/** Sequence number of dbId generation. */
2022-08-10 11:26:08 +00:00
idSeq : 0 ,
2022-08-25 11:39:12 +00:00
/** Map of DB instances to dbId. */
2022-08-10 11:26:08 +00:00
idMap : new WeakMap ,
2022-08-25 11:39:12 +00:00
/** Temp holder for "transferable" postMessage() state. */
xfer : [ ] ,
2022-08-24 05:59:23 +00:00
open : function ( opt ) {
const db = new DB ( opt . filename ) ;
2022-08-10 11:26:08 +00:00
this . dbs [ getDbId ( db ) ] = db ;
if ( ! this . defaultDb ) this . defaultDb = db ;
return db ;
} ,
close : function ( db , alsoUnlink ) {
if ( db ) {
delete this . dbs [ getDbId ( db ) ] ;
2022-08-17 16:44:05 +00:00
const filename = db . fileName ( ) ;
db . close ( ) ;
2022-08-10 11:26:08 +00:00
if ( db === this . defaultDb ) this . defaultDb = undefined ;
2022-08-17 16:44:05 +00:00
if ( alsoUnlink && filename ) {
sqlite3 . capi . sqlite3 _wasm _vfs _unlink ( filename ) ;
}
2022-08-10 11:26:08 +00:00
}
} ,
2022-08-24 18:39:46 +00:00
/ * *
Posts the given worker message value . If xferList is provided ,
it must be an array , in which case a copy of it passed as
postMessage ( ) ' s second argument and xferList . length is set to
0.
* /
2022-08-24 00:10:45 +00:00
post : function ( msg , xferList ) {
2022-08-24 18:39:46 +00:00
if ( xferList && xferList . length ) {
self . postMessage ( msg , Array . from ( xferList ) ) ;
2022-08-10 11:26:08 +00:00
xferList . length = 0 ;
} else {
2022-08-24 00:10:45 +00:00
self . postMessage ( msg ) ;
2022-08-10 11:26:08 +00:00
}
} ,
/** Map of DB IDs to DBs. */
dbs : Object . create ( null ) ,
2022-08-25 11:39:12 +00:00
/ * * F e t c h t h e D B f o r t h e g i v e n i d . T h r o w i f r e q u i r e = t r u e a n d t h e
id is not valid , else return the db or undefined . * /
2022-08-10 11:26:08 +00:00
getDb : function ( id , require = true ) {
return this . dbs [ id ]
|| ( require ? toss ( "Unknown (or closed) DB ID:" , id ) : undefined ) ;
}
} ;
/** Throws if the given db is falsy or not opened. */
const affirmDbOpen = function ( db = wState . defaultDb ) {
return ( db && db . pointer ) ? db : toss ( "DB is not opened." ) ;
} ;
/** Extract dbId from the given message payload. */
const getMsgDb = function ( msgData , affirmExists = true ) {
const db = wState . getDb ( msgData . dbId , false ) || wState . defaultDb ;
return affirmExists ? affirmDbOpen ( db ) : db ;
} ;
const getDefaultDbId = function ( ) {
return wState . defaultDb && getDbId ( wState . defaultDb ) ;
} ;
/ * *
A level of "organizational abstraction" for the Worker
API . Each method in this object must map directly to a Worker
message type key . The onmessage ( ) dispatcher attempts to
dispatch all inbound messages to a method of this object ,
passing it the event . data part of the inbound event object . All
2022-08-24 18:39:46 +00:00
methods must return a plain Object containing any result
2022-08-10 11:26:08 +00:00
state , which the dispatcher may amend . All methods must throw
on error .
* /
const wMsgHandler = {
2022-08-17 16:44:05 +00:00
open : function ( ev ) {
2022-08-24 05:59:23 +00:00
const oargs = Object . create ( null ) , args = ( ev . args || Object . create ( null ) ) ;
2022-08-24 00:10:45 +00:00
if ( args . simulateError ) { // undocumented internal testing option
2022-08-17 16:44:05 +00:00
toss ( "Throwing because of simulateError flag." ) ;
}
2022-08-24 18:39:46 +00:00
const rc = Object . create ( null ) ;
const pDir = sqlite3 . capi . sqlite3 _web _persistent _dir ( ) ;
if ( ! args . filename || ':memory:' === args . filename ) {
oargs . filename = args . filename || '' ;
} else if ( pDir ) {
oargs . filename = pDir + ( '/' === args . filename [ 0 ] ? args . filename : ( '/' + args . filename ) ) ;
2022-08-24 05:59:23 +00:00
} else {
2022-08-24 18:39:46 +00:00
oargs . filename = args . filename ;
}
2022-08-24 00:10:45 +00:00
const db = wState . open ( oargs ) ;
2022-08-24 18:39:46 +00:00
rc . filename = db . filename ;
rc . persistent = ! ! pDir && db . filename . startsWith ( pDir ) ;
rc . dbId = getDbId ( db ) ;
return rc ;
2022-08-17 16:44:05 +00:00
} ,
close : function ( ev ) {
const db = getMsgDb ( ev , false ) ;
const response = {
2022-08-25 11:39:12 +00:00
filename : db && db . filename
2022-08-17 16:44:05 +00:00
} ;
if ( db ) {
2022-08-24 00:10:45 +00:00
wState . close ( db , ( ( ev . args && 'object' === typeof ev . args )
? ! ! ev . args . unlink : false ) ) ;
2022-08-17 16:44:05 +00:00
}
return response ;
} ,
2022-08-25 11:39:12 +00:00
2022-08-10 11:26:08 +00:00
exec : function ( ev ) {
2022-08-24 18:39:46 +00:00
const rc = (
2022-08-24 00:10:45 +00:00
'string' === typeof ev . args
) ? { sql : ev . args } : ( ev . args || Object . create ( null ) ) ;
2022-08-24 20:57:37 +00:00
if ( 'stmt' === rc . rowMode ) {
2022-08-24 18:39:46 +00:00
toss ( "Invalid rowMode for 'exec': stmt mode" ,
2022-08-10 11:26:08 +00:00
"does not work in the Worker API." ) ;
2022-08-25 11:39:12 +00:00
} else if ( ! rc . sql ) {
toss ( "'exec' requires input SQL." ) ;
2022-08-10 11:26:08 +00:00
}
const db = getMsgDb ( ev ) ;
2022-08-24 18:39:46 +00:00
if ( rc . callback || Array . isArray ( rc . resultRows ) ) {
2022-08-10 11:26:08 +00:00
// Part of a copy-avoidance optimization for blobs
2022-08-24 18:39:46 +00:00
db . _blobXfer = wState . xfer ;
2022-08-10 11:26:08 +00:00
}
2022-08-24 20:57:37 +00:00
const theCallback = rc . callback ;
2022-08-24 18:39:46 +00:00
let rowNumber = 0 ;
2022-08-24 20:57:37 +00:00
const hadColNames = ! ! rc . columnNames ;
if ( 'string' === typeof theCallback ) {
if ( ! hadColNames ) rc . columnNames = [ ] ;
2022-08-10 11:26:08 +00:00
/ * T r e a t t h i s a s a w o r k e r m e s s a g e t y p e a n d p o s t e a c h
row as a message of that type . * /
2022-08-24 20:57:37 +00:00
rc . callback = function ( row , stmt ) {
wState . post ( {
type : theCallback ,
columnNames : rc . columnNames ,
rowNumber : ++ rowNumber ,
row : row
} , wState . xfer ) ;
}
2022-08-10 11:26:08 +00:00
}
try {
2022-08-24 18:39:46 +00:00
db . exec ( rc ) ;
if ( rc . callback instanceof Function ) {
2022-08-24 20:57:37 +00:00
rc . callback = theCallback ;
/ * P o s t a s e n t i n e l m e s s a g e t o t e l l t h e c l i e n t t h a t t h e e n d
of the result set has been reached ( possibly with zero
rows ) . * /
wState . post ( {
type : theCallback ,
columnNames : rc . columnNames ,
rowNumber : null /*null to distinguish from "property not set"*/ ,
row : undefined / * undefined because null is a legal row value
for some rowType values , but undefined is not * /
} ) ;
2022-08-24 18:39:46 +00:00
}
} finally {
delete db . _blobXfer ;
2022-08-24 20:57:37 +00:00
if ( rc . callback ) rc . callback = theCallback ;
2022-08-24 18:39:46 +00:00
}
return rc ;
2022-08-10 11:26:08 +00:00
} /*exec()*/ ,
2022-08-25 11:39:12 +00:00
2022-08-24 18:39:46 +00:00
'config-get' : function ( ) {
const rc = Object . create ( null ) , src = sqlite3 . config ;
[
'persistentDirName' , 'bigIntEnabled'
] . forEach ( function ( k ) {
if ( Object . getOwnPropertyDescriptor ( src , k ) ) rc [ k ] = src [ k ] ;
} ) ;
rc . persistenceEnabled = ! ! sqlite3 . capi . sqlite3 _web _persistent _dir ( ) ;
return rc ;
} ,
2022-08-25 11:39:12 +00:00
2022-08-10 11:26:08 +00:00
/ * *
2022-08-24 00:10:45 +00:00
TO ( RE ) DO , once we can abstract away access to the
2022-08-10 11:26:08 +00:00
JS environment ' s virtual filesystem . Currently this
always throws .
Response is ( should be ) an object :
{
buffer : Uint8Array ( db file contents ) ,
filename : the current db filename ,
mimetype : 'application/x-sqlite3'
}
TODO is to determine how / whether this feature can support
exports of ":memory:" and "" ( temp file ) DBs . The latter is
ostensibly easy because the file is ( potentially ) on disk , but
the former does not have a structure which maps directly to a
2022-08-17 16:44:05 +00:00
db file image . We can VACUUM INTO a : memory : / t e m p d b i n t o a
file for that purpose , though .
2022-08-10 11:26:08 +00:00
* /
export : function ( ev ) {
toss ( "export() requires reimplementing for portability reasons." ) ;
2022-08-17 16:44:05 +00:00
/ * *
We need to reimplement this to use the Emscripten FS
interface . That part used to be in the OO # 1 API but that
dependency was removed from that level of the API .
* /
2022-08-10 11:26:08 +00:00
/ * * c o n s t d b = g e t M s g D b ( e v ) ;
const response = {
buffer : db . exportBinaryImage ( ) ,
filename : db . filename ,
mimetype : 'application/x-sqlite3'
} ;
2022-08-24 18:39:46 +00:00
wState . xfer . push ( response . buffer . buffer ) ;
2022-08-10 11:26:08 +00:00
return response ; * * /
} /*export()*/ ,
2022-08-25 11:39:12 +00:00
2022-08-10 11:26:08 +00:00
toss : function ( ev ) {
toss ( "Testing worker exception" ) ;
}
} /*wMsgHandler*/ ;
self . onmessage = function ( ev ) {
ev = ev . data ;
2022-08-24 00:10:45 +00:00
let result , dbId = ev . dbId , evType = ev . type ;
2022-08-10 11:26:08 +00:00
const arrivalTime = performance . now ( ) ;
try {
if ( wMsgHandler . hasOwnProperty ( evType ) &&
wMsgHandler [ evType ] instanceof Function ) {
2022-08-24 00:10:45 +00:00
result = wMsgHandler [ evType ] ( ev ) ;
2022-08-10 11:26:08 +00:00
} else {
toss ( "Unknown db worker message type:" , ev . type ) ;
}
} catch ( err ) {
evType = 'error' ;
2022-08-24 00:10:45 +00:00
result = {
2022-08-24 05:59:23 +00:00
operation : ev . type ,
2022-08-10 11:26:08 +00:00
message : err . message ,
errorClass : err . name ,
input : ev
} ;
if ( err . stack ) {
2022-08-24 00:10:45 +00:00
result . stack = ( 'string' === typeof err . stack )
2022-08-10 11:26:08 +00:00
? err . stack . split ( '\n' ) : err . stack ;
}
if ( 0 ) console . warn ( "Worker is propagating an exception to main thread." ,
2022-08-24 00:10:45 +00:00
"Reporting it _here_ for the stack trace:" , err , result ) ;
2022-08-10 11:26:08 +00:00
}
if ( ! dbId ) {
2022-08-24 00:10:45 +00:00
dbId = result . dbId /*from 'open' cmd*/
2022-08-10 11:26:08 +00:00
|| getDefaultDbId ( ) ;
}
// Timing info is primarily for use in testing this API. It's not part of
// the public API. arrivalTime = when the worker got the message.
2022-08-24 00:10:45 +00:00
wState . post ( {
type : evType ,
dbId : dbId ,
messageId : ev . messageId ,
workerReceivedTime : arrivalTime ,
workerRespondTime : performance . now ( ) ,
departureTime : ev . departureTime ,
2022-08-24 18:39:46 +00:00
// TODO: move the timing bits into...
//timing:{
// departure: ev.departureTime,
// workerReceived: arrivalTime,
// workerResponse: performance.now();
//},
2022-08-24 00:10:45 +00:00
result : result
2022-08-24 18:39:46 +00:00
} , wState . xfer ) ;
2022-08-10 11:26:08 +00:00
} ;
2022-08-24 05:59:23 +00:00
self . postMessage ( { type : 'sqlite3-api' , result : 'worker1-ready' } ) ;
2022-08-22 13:34:13 +00:00
} . bind ( { self , sqlite3 } ) ;
} ) ;