-
Notifications
You must be signed in to change notification settings - Fork 69
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
prombench: Added support for
--bench.version flag.
See prometheus/proposals#41 for rationale. Signed-off-by: bwplotka <[email protected]>
- Loading branch information
Showing
8 changed files
with
215 additions
and
120 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,5 @@ | ||
| INFRA_CMD ?= ../infra/infra | ||
|
|
||
| PROVIDER ?= gke | ||
|
|
||
| .PHONY: deploy clean | ||
| deploy: node_create resource_apply | ||
| # GCP sometimes takes longer than 30 tries when trying to delete nodes | ||
| # if k8s resources are not already cleared | ||
| clean: resource_delete node_delete | ||
| INFRA_CMD ?= ../infra/infra | ||
| PROVIDER ?= gke | ||
|
|
||
| cluster_create: | ||
| ${INFRA_CMD} ${PROVIDER} cluster create -a ${AUTH_FILE} \ | ||
|
|
@@ -37,50 +30,94 @@ cluster_delete: | |
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/cluster_${PROVIDER}.yaml | ||
|
|
||
| # /prombench <...> --bench.directory | ||
| BENCHMARK_DIRECTORY ?= "manifests/prombench" | ||
| # /prombench <...> --bench.version | ||
| BENCHMARK_VERSION ?= "master" | ||
| PROMBENCH_GIT_REPOSITORY ?= [email protected]:prometheus/test-infra.git | ||
| REPO_ROOT_DIR ?= "." | ||
|
|
||
| # maybe_pull_custom_version allows custom benchmarking as designed in | ||
| # https://github.com/prometheus/proposals/pull/41. It allows calling | ||
| # /prombench <release> --bench.version=<@commit or branch> which will cause | ||
| # prombench GH job on Prometheus repo to call infra CLI with the non-master BENCHMARK_VERSION. | ||
| # In such a case we pull a prombench repository for the given branch or commit version | ||
| # and adjust REPO_ROOT_DIR. As a result `make deploy` and `make clean` jobs | ||
| # will apply /manifests/ apply custom manifests or even node pools. | ||
| .PHONE: maybe_pull_custom_version | ||
| maybe_pull_custom_version: | ||
| ifeq (${BENCHMARK_VERSION},"master") | ||
| @echo ">> Using standard benchmark configuration, from the docker image" | ||
| else | ||
| @echo ">> Git pulling custom benchmark configuration from the ${BENCHMARK_VERSION}" | ||
| @$(eval $@_TMP_DIR=$(shell mktemp -d -t "prombench")) | ||
| cd ${$@_TMP_DIR} && git clone ${PROMBENCH_GIT_REPOSITORY} | ||
| ifeq ($(subst @,,${BENCHMARK_VERSION}),${BENCHMARK_VERSION}) | ||
| @echo ">> --bench.version is a branch, reseting to origin/${BENCHMARK_VERSION}" | ||
| cd ${$@_TMP_DIR}/test-infra && git reset --hard origin/${BENCHMARK_VERSION} | ||
| else | ||
| @echo ">> --bench.version is a commit SHA, reseting to $(subst @,,${BENCHMARK_VERSION})" | ||
| cd ${$@_TMP_DIR}/test-infra && git reset --hard $(subst @,,${BENCHMARK_VERSION}) | ||
| endif | ||
| $(eval REPO_ROOT_DIR=${$@_TMP_DIR}/test-infra) | ||
| endif | ||
|
|
||
| .PHONE: clean_tmp_dir | ||
| clean_tmp_dir: # Clean after maybe_pull_custom_version | ||
| [ -z ${maybe_pull_custom_version_TMP_DIR} ] || rm -rf ${maybe_pull_custom_version_TMP_DIR} | ||
|
|
||
| .PHONY: deploy | ||
| deploy: maybe_pull_custom_version node_create resource_apply clean_tmp_dir | ||
|
|
||
| .PHONE: clean | ||
| # GCP sometimes takes longer than 30 tries when trying to delete nodes | ||
| # if k8s resources are not already cleared | ||
| clean: maybe_pull_custom_version resource_delete node_delete clean_tmp_dir | ||
|
|
||
| node_create: | ||
| ${INFRA_CMD} ${PROVIDER} nodes create -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \ | ||
| -v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/prombench/nodes_${PROVIDER}.yaml | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml | ||
|
|
||
| resource_apply: | ||
| $(INFRA_CMD) ${PROVIDER} resource apply -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} \ | ||
| -v PR_NUMBER:${PR_NUMBER} -v RELEASE:${RELEASE} -v DOMAIN_NAME:${DOMAIN_NAME} \ | ||
| -v GITHUB_ORG:${GITHUB_ORG} -v GITHUB_REPO:${GITHUB_REPO} \ | ||
| -f manifests/prombench/benchmark | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/benchmark | ||
|
|
||
| # Required because namespace and cluster-role are not part of the created nodes | ||
| resource_delete: | ||
| $(INFRA_CMD) ${PROVIDER} resource delete -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/prombench/benchmark/1c_cluster-role-binding.yaml \ | ||
| -f manifests/prombench/benchmark/1a_namespace.yaml | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/benchmark/1c_cluster-role-binding.yaml \ | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/benchmark/1a_namespace.yaml | ||
|
|
||
| node_delete: | ||
| $(INFRA_CMD) ${PROVIDER} nodes delete -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \ | ||
| -v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/prombench/nodes_${PROVIDER}.yaml | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml | ||
|
|
||
| all_nodes_running: | ||
| $(INFRA_CMD) ${PROVIDER} nodes check-running -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \ | ||
| -v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/prombench/nodes_${PROVIDER}.yaml | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml | ||
|
|
||
| all_nodes_deleted: | ||
| $(INFRA_CMD) ${PROVIDER} nodes check-deleted -a ${AUTH_FILE} \ | ||
| -v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \ | ||
| -v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \ | ||
| -v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \ | ||
| -v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \ | ||
| -f manifests/prombench/nodes_${PROVIDER}.yaml | ||
| -f ${REPO_ROOT_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| ## Prombench Benchmark Scenario Configuration | ||
|
|
||
| This directory contains resources that are applied (and cleaned) on every benchmark request | ||
| via `infra` CLI using [`make deploy`](../../Makefile) and cleaned using [`make clean`](../../Makefile). | ||
|
|
||
| It assumes running cluster was created via `infra` CLI using `make cluster_create` and `make cluster_delete`. | ||
|
|
||
| ### Variables | ||
|
|
||
| It expects the following templated variables: | ||
|
|
||
| * `.PR_NUMBER`: The PR number from which `/prombench` was triggered. This PR number also tells what commit to use for the `prometheus-test-pr-{{ .PR_NUMBER }}` Prometheus image building (in the init container). | ||
| * `.RELEASE`: The argument provided by `/prombench` caller representing the Prometheus version (docker image tag for `quay.io/prometheus/prometheus:{{ .RELEASE }}`) to compare with, deployed as the `prometheus-test-{{ .RELEASE }}`. | ||
| * `.DOMAIN_NAME` | ||
| * `.LOADGEN_SCALE_UP_REPLICAS` | ||
| * `.GITHUB_ORG` | ||
| * `.GITHUB_REPO` | ||
|
|
||
| ### Customizations | ||
|
|
||
| > NOTE: See https://github.com/prometheus/proposals/pull/41 for design. | ||
| On the `master` branch, in this directory, we maintain the standard, single benchmarking scenario used | ||
| as an acceptance validation for Prometheus. It's important to ensure it represents common Prometheus configuration. | ||
|
|
||
| The only user related parameter for the standard scenario is `RELEASE` version. | ||
|
|
||
| However, it's possible to create, a fully custom benchmarking scenarios for `/prombench` via `--bench.version=<branch|@commit>` flag. | ||
|
|
||
| Here are an example steps: | ||
|
|
||
| 1. Create a new branch on https://github.com/prometheus/test-infra e.g. `benchmark/scenario1`. | ||
| 2. Modify this directory to your liking e.g. changing query load, metric load of advanced Prometheus configuration. It's also possible to make Prometheus deployments and versions exactly the same, but vary in a single configuration flag, for feature benchmarking. | ||
|
|
||
| > WARN: When customizing this directory, don't change `1a_namespace.yaml` or `1c_cluster-role-binding.yaml` filenames as they are used for cleanup routine. Or, if you change it, know what you're doing in relation to [`make clean` job](../../Makefile). | ||
| 3. Push changes to the new branch. | ||
| 4. From the Prometheus PR comment, call prombench as `/prombench <release> --bench.version=benchmark/scenario1` or `/prombench <release> --bench.version=@<relevant commit SHA from the benchmark/scenario1>` to use configuration files from this custom branch. | ||
|
|
||
| Other details: | ||
|
|
||
| * Other custom branch modifications other than to this directory do not affect prombench (e.g. to infra CLI or makefiles). | ||
| * `--bench.version` is designed for a short-term or even one-off benchmark scenario configurations. It's not designed for long-term, well maintained scenarios. For the latter reason we can later e.g. maintain multiple `manifests/prombench` directories and introduce a new `--bench.directory` flag. | ||
| * Non-maintainers can follow similar process, but they will need to ask maintainer for a new branch and PR review. We can consider extending `--bench.version` to support remote repositories if this becomes a problem. | ||
| * Custom benchmarking logic is implemented in the [`maybe_pull_custom_version` make job](../../Makefile) and invoked by the prombench GH job on Prometheus repo on `deploy` and `clean`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.