Skip to content

Latest commit

 

History

History
154 lines (122 loc) · 7.56 KB

README.md

File metadata and controls

154 lines (122 loc) · 7.56 KB
Raw

A collection of benchmarking tools and libraries for testing,
comparing, and improving the performance of SurrealDB.


     

Blog   Github   LinkedIn   Twitter   Youtube   Dev   Discord   StackOverflow


Benchmarks

This repository exists as a central repository for a collection of benchmarking tools and libraries for testing, comparing, and improving the performance of SurrealDB. Each tool linked to from this location exists within its own repository.

As we add support for additional benchmarking suites, we will add and link to them in this repository.

crud-bench

The crud-bench benchmarking tool is an open-source benchmarking tool for testing and comparing the performance of a number of different workloads on embedded, networked, and remote databases. It can be used to compare both SQL and NoSQL platforms including key-value, and embedded databases. Importantly crud-bench focuses on testing additional features which are not present in other benchmarking tools, but which are available in SurrealDB.

The primary purpose of crud-bench is to continually test and monitor the performance of features and functionality built in to SurrealDB, enabling developers working on features in SurrealDB to assess the impact of their changes on database queries and performance.

The crud-bench benchmarking tool is being actively developed with new features and functionality being added regularly.

Getting started with the crud-bench benchmarking tool
  1. Navigate to the crud-bench repository.
git clone -b main https://github.com/surrealdb/crud-bench
  1. Build the crud-bench tool in release mode
cargo build --release && ln -s target/release/crud-bench crud-bench
  1. View the configuration options
crud-bench --help
  1. Ensure the Docker daemon is running
dockerd
  1. Run a local benchmark with custom options
crud-bench -d surrealdb-rocksdb -s 5000000 -c 128 -t 48 -k string26 -r
  1. Run a remote benchmark against a remote SurrealDB server
crud-bench -d surrealdb --endpoint ws://127.0.0.1:8000 -s 5000000 -c 128 -t 48 -k string26 -r

ann-benchmarks

The Approximate Nearest Neighbour benchmarks suite is a testing tool for comparing the performance of vector datasets, with a focus on approximate and exact nearest neighbour algorithms across different data platforms, databases, and libraries. This project contains tools to benchmark various implementations of approximate nearest neighbor, and exact nearest neighbour search for selected metrics - with pre-generated datasets, and a test suite to verify function integrity.

Getting started with the ann-benchmarks benchmarking tool
  1. Navigate to the surrealdb branch on the repository.
git clone -b surrealdb https://github.com/surrealdb/ann-benchmarks
  1. Setup a Python virtual environment
python3.10 -m venv path/to/venv && source path/to/venv/bin/activate
  1. Install the benchmark requirements
pip3.10 install -r requirements.txt
  1. Install the desired benchmark algorithm
python3.10 install.py --algorithm surreal_hnsw
  1. Run a specific benchmark with a specified dataset
python3.10 run.py --run-disabled --algorithm surreal_hnsw --dataset random-xs-20-euclidean

go-ycsb

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare the relative performance of NoSQL database management systems. This Golang port of the original benchmarking suite allows benchmarking comparison using the Golang programing language, and is sponsored by Pingcap.

Getting started with the go-ycsb benchmarking tool
  1. Navigate to the surrealdb branch on the repository.
git clone -b surrealdb https://github.com/surrealdb/go-ycsb
  1. Build the go-ycsb binary
make quick
  1. Test a workload locally against SurrealDB
./bin/go-ycsb load surrealdb -P workloads/workloada
./bin/go-ycsb run surrealdb -P workloads/workloada
  1. Test a workload against a remote SurrealDB server
./bin/go-ycsb load surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root
./bin/go-ycsb run surrealdb -P workloads/workloada -p surrealdb.uri='ws://127.0.0.1:8000' -p surrealdb.user=root -p surrealdb.pass=root

nosqlbench

Important

The SurrealDB changes to this benchmarking tool have not yet been released. Please check back later.

NoSQLBench is a serious performance testing tool for the NoSQL ecosystem, sponsored by DataStax. It brings together features and capabilities that are not found in any other tool. The core machinery of NoSQLBench has been built with attention to detail. It has been battle tested within DataStax and in the NoSQL ecosystem as a way to help users validate their data models, baseline system performance, and qualify system designs for scale.