A collection of benchmarking tools and libraries for testing,
comparing, and improving the performance of SurrealDB.

     

             

Benchmarks

This repository exists as a central repository for a collection of benchmarking tools and libraries for testing, comparing, and improving the performance of SurrealDB. Each tool linked to from this location exists within its own repository.

As we add support for additional benchmarking suites, we will add and link to them in this repository.

crud-bench

The crud-bench benchmarking tool is an open-source benchmarking tool for testing and comparing the performance of a number of different workloads on embedded, networked, and remote databases. It can be used to compare both SQL and NoSQL platforms including key-value, and embedded databases. Importantly crud-bench focuses on testing additional features which are not present in other benchmarking tools, but which are available in SurrealDB.

The primary purpose of crud-bench is to continually test and monitor the performance of features and functionality built in to SurrealDB, enabling developers working on features in SurrealDB to assess the impact of their changes on database queries and performance.

The crud-bench benchmarking tool is being actively developed with new features and functionality being added regularly.

Getting started with the crud-bench benchmarking tool

1. Navigate to the crud-bench repository.

git clone -b main https://github.com/surrealdb/crud-bench

2. Build the crud-bench tool in release mode

cargo build --release && ln -s target/release/crud-bench crud-bench

3. View the configuration options

crud-bench --help

4. Ensure the Docker daemon is running

dockerd

5. Run a local benchmark with custom options

crud-bench -d surrealdb-rocksdb -s 5000000 -c 128 -t 48 -k string26 -r

6. Run a remote benchmark against a remote SurrealDB server

crud-bench -d surrealdb --endpoint ws://127.0.0.1:8000 -s 5000000 -c 128 -t 48 -k string26 -r

ann-benchmarks

The Approximate Nearest Neighbour benchmarks suite is a testing tool for comparing the performance of vector datasets, with a focus on approximate and exact nearest neighbour algorithms across different data platforms, databases, and libraries. This project contains tools to benchmark various implementations of approximate nearest neighbor, and exact nearest neighbour search for selected metrics - with pre-generated datasets, and a test suite to verify function integrity.

Getting started with the ann-benchmarks benchmarking tool

1. Navigate to the surrealdb branch on the repository.

git clone -b surrealdb https://github.com/surrealdb/ann-benchmarks

2. Setup a Python virtual environment

python3.10 -m venv path/to/venv && source path/to/venv/bin/activate

3. Install the benchmark requirements

pip3.10 install -r requirements.txt

4. Install the desired benchmark algorithm

python3.10 install.py --algorithm surreal_hnsw

5. Run a specific benchmark with a specified dataset

python3.10 run.py --run-disabled --algorithm surreal_hnsw --dataset random-xs-20-euclidean

go-ycsb

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare the relative performance of NoSQL database management systems. This Golang port of the original benchmarking suite allows benchmarking comparison using the Golang programing language, and is sponsored by Pingcap.

Getting started with the go-ycsb benchmarking tool

1. Navigate to the surrealdb branch on the repository.

git clone -b surrealdb https://github.com/surrealdb/go-ycsb

2. Build the go-ycsb binary

make quick

3. Test a workload locally against SurrealDB

./bin/go-ycsb load surrealdb -P workloads/workloada
./bin/go-ycsb run surrealdb -P workloads/workloada

4. Test a workload against a remote SurrealDB server

./bin/go-ycsb load surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root
./bin/go-ycsb run surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root

nosqlbench

Important

The SurrealDB changes to this benchmarking tool have not yet been released. Please check back later.

NoSQLBench is a serious performance testing tool for the NoSQL ecosystem, sponsored by DataStax. It brings together features and capabilities that are not found in any other tool. The core machinery of NoSQLBench has been built with attention to detail. It has been battle tested within DataStax and in the NoSQL ecosystem as a way to help users validate their data models, baseline system performance, and qualify system designs for scale.