bolt

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL. Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values. That's it. Project Status Bolt is stable, the API is fixed, and the file format is fixed. Full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. Bolt is currently used in high-load production environments serving databases as large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed services every day. A message from the author The original goal of Bolt was to provide a simple pure Go key/value store and to not bloat the code with extraneous features. To that end, the project has been a success. However, this limited scope also means that the project is complete. Maintaining an open source database requires an immense amount of time and energy. Changes to the code can have unintended and sometimes catastrophic effects so even simple changes require hours and hours of careful testing and validation. Unfortunately I no longer have the time or energy to continue this work. Bolt is in a stable state and has years of successful production use. As such, I feel that leaving it in its current state is the most prudent course of action. If you are interested in using a more featureful version of Bolt, I suggest that you look at the CoreOS fork called bbolt. Ben Johnson (@benbjohnson) Table of Contents Getting Started Installing Opening a database Transactions Read-write transactions Read-only transactions Batch read-write transactions Managing transactions manually Using buckets Using key/value pairs Autoincrementing integer for the bucket Iterating over keys Prefix scans Range scans ForEach() Nested buckets Database backups Statistics Read-Only Mode Mobile Use (iOS/Android) Resources Comparison with other databases Postgres, MySQL, & other relational databases LevelDB, RocksDB LMDB Caveats & Limitations Reading the Source Other Projects Using Bolt Getting Started Installing To start using Bolt, install Go and run go get: $ go get github.com/boltdb/bolt/... This will retrieve the library and install the bolt command line utility into your $GOBIN path. Opening a database The top-level object in Bolt is a DB. It is represented as a single file on your disk and represents a consistent snapshot of your data. To open your database, simply use the bolt.Open() function: package main import ( "log" "github.com/boltdb/bolt" ) func main() { // Open the my.db data file in your current directory. // It will be created if it doesn't exist. db, err := bolt.Open("my.db", 0600, nil) if err != nil { log.Fatal(err) } defer db.Close() ... } Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it. To prevent an indefinite wait you can pass a timeout option to the Open() function: db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second}) Transactions Bolt allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started. Individual transactions and all objects created from them (e.g. buckets, keys) are not thread safe. To work with data in multiple goroutines you must start a transaction for each one or use locking to ensure only one goroutine accesses a transaction at a time. Creating transaction from the DB is thread safe. Read-only transactions and read-write transactions should not depend on one another and generally shouldn't be opened simultaneously in the same goroutine. This can cause a deadlock as the read-write transaction needs to periodically re-map the data file but it cannot do so while a read-only transaction is open. Read-write transactions To start a read-write transaction, you can use the DB.Update() function: err := db.Update(func(tx *bolt.Tx) error { ... return nil }) Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction. Always check the return error as it will report any disk failures that can cause your transaction to not complete. If you return an error within your closure it will be passed through. Read-only transactions To start a read-only transaction, you can use the DB.View() function: err := db.View(func(tx *bolt.Tx) error { ... return nil }) You also get a consistent view of the database within this closure, however, no mutating operations are allowed within a read-only transaction. You can only retrieve buckets, retrieve values, and copy the database within a read-only transaction. Batch read-write transactions Each DB.Update() waits for disk to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function: err := db.Batch(func(tx *bolt.Tx) error { ... return nil }) Concurrent Batch calls are opportunistically combined into larger transactions. Batch is only useful when there are multiple goroutines calling it. The trade-off is that Batch can call the given function multiple times, if parts of the transaction fail. The function must be idempotent and side effects must take effect only after a successful return from DB.Batch(). For example: don't display messages from inside the function, instead set variables in the enclosing scope: var id uint64 err := db.Batch(func(tx *bolt.Tx) error { // Find last