better-sqlite3

I’ve enjoyed https://github.com/WiseLibs/better-sqlite3/blob/master/docs/... as a code of conduct. Simple, well-motivated, well-practiced (I think). Unlike more convoluted CoCs, it’s tightly scoped on work, not fun. Specific labor intended to produce contributions instead of enjoyable social experiences. I feel a lot more OSS should avoid fun. Fun doesn’t seem to scale, especially when it gets in the way of real work.

One huge benefit of using SQLite over a traditional server/client DBMS is the ability to easily add SQL functions that call into your host language and memory-space. For example, we’re using better-sqlite3 which has a convenient API for adding SQL functions [1], and we have dozens of helper methods for dealing with time using the temporal-polyfill module. We have custom JSON-based serialisation formats for PlainDate, PlainTime, PlainDateTime, ZonedDateTome, etc. Then in SQL we can call ‘isDate_Between__(a, b, c)`. a, b, and c are deserialised by the JS (TS) function, the logic is run, and the result is returned to SQLite. We’ve had no performance issues with this approach, though we’re only dealing with simple CRUD stuff. No big data. You can even use these functions with generated columns and indexes, but I haven’t found a proper use for

These may be useful reading: https://github.com/TryGhost/node-sqlite3/issues/408#issue-57... https://github.com/WiseLibs/better-sqlite3/issues/32#issueco... Copying a quote from the second: > The sqlite3 C API serializes all operations (even reads) within a single process. You can parallelize reads to the database but only by having multiple processes, in which case one process being blocked doesn't affect the other processes anyways. In other words, because sqlite3 serializes everything, doing things asynchronously won't speed up database access within a process. It would only free up time for your app to do other things (like HTTP requests to other servers). Unfortunately, the overhead imposed on sqlite3 to serialize asynchronous operations is quite high, making it disadvantageous 95% of the time.

If I were charged with this task I'd probably take an actual SQLite DB and write methods to parse data from text files, then process it within SQLite, and serialize it back to the text file when an API method is called or after each processing step. A fresh DB engine, fully re-implemented to match SQLite's API is insane unless you're a prodigy or have a sizable team and will come with numerous downsides. I can recommend https://github.com/WiseLibs/better-sqlite3 and https://github.com/loveencounterflow/dbay

better-sqlite3 is orders of magnitude faster than the async SQLite bindings. We found this to be true when testing SQLite options for Notion's desktop app anyways. The "why should I use this" bits sound boastful but are reasonable. https://github.com/WiseLibs/better-sqlite3#why-should-i-use-...

Why do you have async reads and writes? There's no client-server setup here, using async / await just introduces pointless waiting. https://github.com/WiseLibs/better-sqlite3

Kent's recommended NodeJS module, `better-sqlite3`, has some very nifty features including the creation of JavaScript user-defined functions[0] that (if I understand this right) can be called from SQLite. Combined with TRIGGERs, I wonder if it might fire a function within the app when an UPDATE/INSERT happens from a different process? (This is me wondering out loud, I don't actually know.) I also recommend checking out Replicache[1] and alternatives, which may be a better way to handle the networking and database replication so that it doesn't rely on the underlying DB. [0] https://github.com/WiseLibs/better-sqlite3/blob/HEAD/docs/ap... [1] https://replicache.dev/

The frontend is in Vue. Both the FE and BE are in TypeScript. Parallelism is provided by https://github.com/photostructure/batch-cluster.js/ Metadata reads and writes are via https://github.com/photostructure/exiftool-vendored.js/ Persistence is via SQLite: https://github.com/WiseLibs/better-sqlite3 Image transforms are via Sharp: https://sharp.pixelplumbing.com/ My more nerdier blog posts are tagged here: https://photostructure.com/tags/coding/

It would be great to have some benchmarks against better-sqlite and the regular sqlite libraries, like in https://github.com/WiseLibs/better-sqlite3

I tried to replace SQLite with DuckDB for a customized install of better-sqlite3[1] and failed. [1] https://github.com/JoshuaWise/better-sqlite3

Best option for SQlite with node is this. https://github.com/JoshuaWise/better-sqlite3 Author is all over the issues section, and seems very knowledgeable about how SQLite works.