Integrating block-sync in an init-container #779

kushalShukla-web · 2024-11-08T16:46:07Z

Changes that are introduced in this pr , followed by Create a CLI for uploading and downloading blocks #760 .
Integrated download-key in an init container that downloads the key.yml file from the Prometheus repository. It uses the key.sh script to fetch the pull request and copy the key.yml file into the volumeMount section:

volumeMounts:
  - name: key
    mountPath: /config

Integrated data-downloaderon both pr and released side, which uses the block-sync binary to download pre-generated block data.
Updated the prometheus-builder image, modifying the builder.sh file to copy the key.yml file into the volumeMount section so that the PR section can also download the pre-generated block data.

bboreham

I'm struggling to find my way around. Some trivial points stand out, but I need better orientation.

removed from block-sync:

Please explain each PR in terms of what it does, not what some other PR didn't do.

For example, you could describe the intended purpose of key.sh.

bboreham · 2024-11-12T09:18:18Z

prombench/docs/kind.md

+- **Option 2: Skip downloading data**
+
+   If you don’t need to download data, edit the `3b_prometheus-test_deployment.yaml` file:


Could we instead tell people to create an empty Secret?

bboreham · 2024-11-12T09:18:35Z

prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml

@@ -34,7 +34,7 @@ spec:
        runAsUser: 0
      initContainers:
      - name: prometheus-builder
-        image: docker.io/prominfra/prometheus-builder:master
+        image: kushalshukla/builder


Fix up before merge.

bboreham · 2024-11-12T09:19:08Z

tools/prometheus-builder/build.sh

@@ -6,7 +6,7 @@ if [[ -z $PR_NUMBER || -z $VOLUME_DIR || -z $GITHUB_ORG || -z $GITHUB_REPO ]]; t
    echo "ERROR:: environment variables not set correctly"
    exit 1;
 fi
-
+ 


Spurious change

bboreham · 2024-11-12T09:19:16Z

tools/prometheus-builder/Dockerfile

 RUN chmod +x /go/src/github.com/build.sh

-ENTRYPOINT ["/go/src/github.com/build.sh"]
+ENTRYPOINT ["/go/src/github.com/build.sh"]


Spurious change

bboreham · 2024-11-12T09:23:30Z

tools/prometheus-builder/Dockerfile

@@ -4,6 +4,8 @@ RUN mkdir -p /go/src/github.com

 COPY ./build.sh /go/src/github.com/build.sh

+COPY ./key.sh /download-key/key.sh


Why is this in such a different place from the other script?

bboreham · 2024-11-12T09:24:24Z

tools/prometheus-builder/build.sh

+# emptyDir. This file will later be used by the data-downloader init container.
+MKDIR="/config"
+if [ -f "$DIR/key.yml" ]; then
+    echo "File exists."


Who is expected to read this message, and what are they expected to do about it?

bboreham · 2024-11-12T09:26:06Z

prombench/docs/kind.md

+2. Before applying benchmarking objects , You have two choices to make:  
+- **Option 1: Download data from object storage**


How would someone coming to this for the first time understand this choice?
Can we give some context why data might be downloaded?

kakkoyun · 2024-11-13T09:52:36Z

tools/prometheus-builder/build.sh

+# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container, 
+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+MKDIR="/config"


I think you need to change the name.

kakkoyun · 2024-11-13T09:59:22Z

tools/prometheus-builder/build.sh

@@ -24,6 +24,17 @@ fi

 git checkout pr-branch

+# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container, 


It would be better to have this in another script.

from block-sync. Signed-off-by: Kushal Shukla <[email protected]>

2. Modified Prometheus deployment yml file 3. changed Prometheus Builder Docker file Signed-off-by: Kushal Shukla <[email protected]>

…nfig file is empty. Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>

prombench/docs/eks.md

kakkoyun · 2024-11-20T11:14:28Z

prombench/docs/eks.md

+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml```  with the necessary credentials as per your object storage. This secret enables access to the stored data.


Suggested change

To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml``` with the necessary credentials as per your object storage. This secret enables access to the stored data.

To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data.

prombench/docs/eks.md

prombench/docs/gke.md

kakkoyun · 2024-11-20T11:20:33Z

tools/prometheus-builder/build.sh

+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."
+    cp  "$DIR/key.yml" "$STORAGE/key.yml"
+else
+    echo "INFO:: key.yml File does not exist on $DIR/key.yml directory."


We should add more context or remove this line. e.g. File does not exists so we won't download XXX

kakkoyun · 2024-11-20T11:21:36Z

tools/prometheus-builder/build.sh

+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+if [ -f "$DIR/key.yml" ]; then
+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."


"It exists so we will download the blocks from XXX."

Signed-off-by: Kushal Shukla <[email protected]>

Removed the code for exiting container when secret is not present. Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web changed the title ~~Added yaml file changes and prometheus builder changes that was removed~~ changes that are not included in Block-sync Nov 8, 2024

kushalShukla-web force-pushed the block-yml branch from 94a489b to 85e464f Compare November 8, 2024 17:02

kushalShukla-web changed the title ~~changes that are not included in Block-sync~~ Integrating Block-sync in an Init-Container. Nov 9, 2024

bboreham reviewed Nov 12, 2024

View reviewed changes

kushalShukla-web force-pushed the block-yml branch 6 times, most recently from 65358a1 to b18501c Compare November 13, 2024 07:10

kakkoyun changed the title ~~Integrating Block-sync in an Init-Container.~~ Integrating Block-sync in an init-container Nov 13, 2024

kakkoyun reviewed Nov 13, 2024

View reviewed changes

kakkoyun changed the title ~~Integrating Block-sync in an init-container~~ Integrating block-sync in an init-container Nov 13, 2024

kushalShukla-web force-pushed the block-yml branch 3 times, most recently from 42e7b36 to 1dfe367 Compare November 14, 2024 12:49

kushalShukla-web added 2 commits November 15, 2024 12:39

Added yaml file changes and prometheus builder changes that was removed

e55d9dd

from block-sync. Signed-off-by: Kushal Shukla <[email protected]>

1. Changed Image name

beec55f

2. Modified Prometheus deployment yml file 3. changed Prometheus Builder Docker file Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch 4 times, most recently from 8197ba6 to 1dbaaf1 Compare November 15, 2024 07:20

Added empty secretes ,modified newstore function to normal stop if co…

1c39d4f

…nfig file is empty. Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch 2 times, most recently from aab03cb to 18b922f Compare November 19, 2024 15:05

Added --storage.tsdb.retention.size to retain data for certain size.

2c1f884

Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch from 18b922f to 2c1f884 Compare November 19, 2024 15:08

kakkoyun reviewed Nov 20, 2024

View reviewed changes

prombench/docs/gke.md Show resolved Hide resolved

kakkoyun reviewed Nov 20, 2024

View reviewed changes

kushalShukla-web force-pushed the block-yml branch from 93ba277 to b795fea Compare November 21, 2024 13:29

minior changes

3e834aa

Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch 3 times, most recently from b405f52 to 0f98ebd Compare November 21, 2024 13:49

Removed Image Building from config.yml file.

07476cd

Removed the code for exiting container when secret is not present. Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch 2 times, most recently from ec570f4 to 905b6b3 Compare November 22, 2024 15:06

Replaced key.yml with object-config.yml file.

23a7a07

Signed-off-by: Kushal Shukla <[email protected]>

kushalShukla-web force-pushed the block-yml branch from 905b6b3 to 23a7a07 Compare November 22, 2024 20:32

kushalShukla-web mentioned this pull request Dec 14, 2024

Added Bucket-config as a configmap in a Block-sync init container. #810

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrating block-sync in an init-container #779

Integrating block-sync in an init-container #779

kushalShukla-web commented Nov 8, 2024 •

edited

Loading

bboreham left a comment

bboreham Nov 12, 2024

bboreham Nov 12, 2024

bboreham Nov 12, 2024

bboreham Nov 12, 2024

bboreham Nov 12, 2024

bboreham Nov 12, 2024

bboreham Nov 12, 2024

kakkoyun Nov 13, 2024

kakkoyun Nov 13, 2024

kakkoyun Nov 20, 2024

kakkoyun Nov 20, 2024

kakkoyun Nov 20, 2024 •

edited

Loading

		- Option 2: Skip downloading data

		If you don’t need to download data, edit the `3b_prometheus-test_deployment.yaml` file:

		@@ -4,6 +4,8 @@ RUN mkdir -p /go/src/github.com

		COPY ./build.sh /go/src/github.com/build.sh

		COPY ./key.sh /download-key/key.sh

		2. Before applying benchmarking objects , You have two choices to make:
		- Option 1: Download data from object storage

		@@ -24,6 +24,17 @@ fi

		git checkout pr-branch

		# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container,

	To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml``` with the necessary credentials as per your object storage. This secret enables access to the stored data.
	To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data.

Integrating block-sync in an init-container #779

Are you sure you want to change the base?

Integrating block-sync in an init-container #779

Conversation

kushalShukla-web commented Nov 8, 2024 • edited Loading

bboreham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kakkoyun Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

kushalShukla-web commented Nov 8, 2024 •

edited

Loading

kakkoyun Nov 20, 2024 •

edited

Loading