neon

Neon Neon is an open-source serverless Postgres database platform. It separates storage and compute and substitutes the PostgreSQL storage layer by redistributing data across a cluster of nodes. Quick start Try the Neon Free Tier to create a serverless Postgres instance. Then connect to it with your preferred Postgres client (psql, dbeaver, etc) or use the online SQL Editor. See Connect from any application for connection instructions. Alternatively, compile and run the project locally. Architecture overview A Neon installation consists of compute nodes and the Neon storage engine. Compute nodes are stateless PostgreSQL nodes backed by the Neon storage engine. The Neon storage engine consists of two major components: Pageserver: Scalable storage backend for the compute nodes. Safekeepers: The safekeepers form a redundant WAL service that received WAL from the compute node, and stores it durably until it has been processed by the pageserver and uploaded to cloud storage. See developer documentation in SUMMARY.md for more information. Running a local development environment Neon can be run on a workstation for small experiments and to test code changes, by following these instructions. Installing dependencies on Linux Install build dependencies and other applicable packages On Ubuntu or Debian, this set of packages should be sufficient to build the code: apt install build-essential libtool libreadline-dev zlib1g-dev flex bison libseccomp-dev \ libssl-dev clang pkg-config libpq-dev cmake postgresql-client protobuf-compiler \ libprotobuf-dev libcurl4-openssl-dev openssl python3-poetry lsof libicu-dev On Fedora, these packages are needed: dnf install flex bison readline-devel zlib-devel openssl-devel \ libseccomp-devel perl clang cmake postgresql postgresql-contrib protobuf-compiler \ protobuf-devel libcurl-devel openssl poetry lsof libicu-devel libpq-devel python3-devel \ libffi-devel On Arch based systems, these packages are needed: pacman -S base-devel readline zlib libseccomp openssl clang \ postgresql-libs cmake postgresql protobuf curl lsof Building Neon requires 3.15+ version of protoc (protobuf-compiler). If your distribution provides an older version, you can install a newer version from here. Install Rust # recommended approach from https://www.rust-lang.org/tools/install curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Installing dependencies on macOS (12.3.1) Install XCode and dependencies xcode-select --install brew install protobuf openssl flex bison icu4c pkg-config m4 # add openssl to PATH, required for ed25519 keys generation in neon_local echo 'export PATH="$(brew --prefix openssl)/bin:$PATH"' >> ~/.zshrc If you get errors about missing m4 you may have to install it manually: brew install m4 brew link --force m4 Install Rust # recommended approach from https://www.rust-lang.org/tools/install curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Install PostgreSQL Client # from https://stackoverflow.com/questions/44654216/correct-way-to-install-psql-without-full-postgres-on-macos brew install libpq brew link --force libpq Rustc version The project uses rust toolchain file to define the version it's built with in CI for testing and local builds. This file is automatically picked up by rustup that installs (if absent) and uses the toolchain version pinned in the file. rustup users who want to build with another toolchain can use the rustup override command to set a specific toolchain for the project's directory. non-rustup users most probably are not getting the same toolchain automatically from the file, so are responsible to manually verify that their toolchain matches the version in the file. Newer rustc versions most probably will work fine, yet older ones might not be supported due to some new features used by the project or the crates. Building on Linux Build neon and patched postgres # Note: The path to the neon sources can not contain a space. git clone --recursive https://github.com/neondatabase/neon.git cd neon # The preferred and default is to make a debug build. This will create a # demonstrably slower build than a release build. For a release build, # use "BUILD_TYPE=release make -j`nproc` -s" # Remove -s for the verbose build log make -j`nproc` -s Building on OSX Build neon and patched postgres # Note: The path to the neon sources can not contain a space. git clone --recursive https://github.com/neondatabase/neon.git cd neon # The preferred and default is to make a debug build. This will create a # demonstrably slower build than a release build. For a release build, # use "BUILD_TYPE=release make -j`sysctl -n hw.logicalcpu` -s" # Remove -s for the verbose build log make -j`sysctl -n hw.logicalcpu` -s Dependency installation notes To run the psql client, install the postgresql-client package or modify PATH and LD_LIBRARY_PATH to include pg_install/bin and pg_install/lib, respectively. To run the integration tests or Python scripts (not required to use the code), install Python (3.11 or higher), and install the python3 packages using ./scripts/pysync (requires poetry>=1.8) in the project directory. Running neon database Start pageserver and postgres on top of it (should be called from repo root): # Create repository in .neon with proper paths to binaries and data # Later that would be responsibility of a package install script > cargo neon init Initializing pageserver node 1 at '127.0.0.1:64000' in ".neon" # start pageserver, safekeeper, and broker for their intercommunication > cargo neon start Starting neon broker at 127.0.0.1:50051. storage_broker started, pid: 2918372 Starting pageserver node 1 at '127.0.0.1:64000' in ".neon". pageserver started, pid: 2918386 Starting safekeeper at '127.0.0.1:5454' in '.neon/safekeepers/sk1'. safekeeper 1 started, pid: 2918437 # create initial tenant and use it as a default for every future neon_local invocation > cargo neon tenant create --set-default tenant 9