Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tolerations are not parsed correctly for Kubernetes driver when appending nodes #2856

Open
3 tasks done
aiell0 opened this issue Dec 12, 2024 · 1 comment
Open
3 tasks done

Tolerations are not parsed correctly for Kubernetes driver when appending nodes #2856

aiell0 opened this issue Dec 12, 2024 · 1 comment
Labels
area/driver/kubernetes

Comments

@aiell0
Copy link

aiell0 commented Dec 12, 2024 •
edited
Loading

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

I am attempting to append builder nodes to my buildx kubernetes driver. I have taints on the Kubernetes compute that I wish to tolerate. Upon following the docs, I could not get it to work.

Expected behaviour

docker buildx create --debug --name charming_chaplygin --append --node buildx-arm64 --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 --driver-opt namespace=buildx --driver-opt requests.cpu=2 --driver-opt requests.memory=4G --driver-opt requests.ephemeral-storage=3Gi --driver-opt nodeselector=gha=true --driver-opt tolerations=key=build-agent,value=true --driver-opt annotations=karpenter.sh/do-not-disrupt=true --platform linux/arm/v7
WARNING: new settings may not be used until builder is restarted
charming_chaplygin

Actual behaviour

docker buildx create --debug --name charming_chaplygin --append --node buildx-arm64 --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 --driver-opt namespace=buildx --driver-opt requests.cpu=2 --driver-opt requests.memory=4G --driver-opt requests.ephemeral-storage=3Gi --driver-opt nodeselector=gha=true --driver-opt tolerations=key=build-agent,value=true --driver-opt annotations=karpenter.sh/do-not-disrupt=true --platform linux/arm/v7
WARNING: new settings may not be used until builder is restarted
ERROR: failed to initialize builder charming_chaplygin (buildx-arm64): invalid driver option value for driver kubernetes

Buildx version

github.com/docker/buildx v0.18.0 fa4461b

Docker info

Client:
 Version:    27.3.1
 Context:    orbstack
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0
    Path:     /Users/lawrenceaiello/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /Users/lawrenceaiello/.docker/cli-plugins/docker-compose

Server:
 Containers: 21
  Running: 5
  Paused: 0
  Stopped: 16
 Images: 11
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 207ad711eabd375a01713109a8a197d197ff6542
 runc version: 7cb363254b69e10320360b63fb73e0ffb5da7bf2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.10-orbstack-00282-g72f45320fe21
 Operating System: OrbStack
 OSType: linux
 Architecture: aarch64
 CPUs: 11
 Total Memory: 8.78GiB
 Name: orbstack
 ID: a631c4fe-775f-4ed4-8180-8e9e18f54bda
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 192.168.97.0/24, Size: 24
   Base: 192.168.107.0/24, Size: 24
   Base: 192.168.117.0/24, Size: 24
   Base: 192.168.147.0/24, Size: 24
   Base: 192.168.148.0/24, Size: 24
   Base: 192.168.155.0/24, Size: 24
   Base: 192.168.156.0/24, Size: 24
   Base: 192.168.158.0/24, Size: 24
   Base: 192.168.163.0/24, Size: 24
   Base: 192.168.164.0/24, Size: 24
   Base: 192.168.165.0/24, Size: 24
   Base: 192.168.166.0/24, Size: 24
   Base: 192.168.167.0/24, Size: 24
   Base: 192.168.171.0/24, Size: 24
   Base: 192.168.172.0/24, Size: 24
   Base: 192.168.181.0/24, Size: 24
   Base: 192.168.183.0/24, Size: 24
   Base: 192.168.186.0/24, Size: 24
   Base: 192.168.207.0/24, Size: 24
   Base: 192.168.214.0/24, Size: 24
   Base: 192.168.215.0/24, Size: 24
   Base: 192.168.216.0/24, Size: 24
   Base: 192.168.223.0/24, Size: 24
   Base: 192.168.227.0/24, Size: 24
   Base: 192.168.228.0/24, Size: 24
   Base: 192.168.229.0/24, Size: 24
   Base: 192.168.237.0/24, Size: 24
   Base: 192.168.239.0/24, Size: 24
   Base: 192.168.242.0/24, Size: 24
   Base: 192.168.247.0/24, Size: 24

Builders list

NAME/NODE                 DRIVER/ENDPOINT                                                                                                   STATUS     BUILDKIT   PLATFORMS
charming_chaplygin        kubernetes                                                                                                                              
 \_ charming_chaplygin0    \_ kubernetes:///charming_chaplygin?deployment=buildkit-2fe904ae-c5bf-4dcd-8313-552b1c79985b-91728&kubeconfig=   inactive              
 \_ buildx-arm64           \_ kubernetes:///charming_chaplygin?deployment=buildx-arm64&kubeconfig=                                          inactive              linux/arm/v7*
container                 docker-container                                                                                                                        
 \_ container0             \_ orbstack                                                                                                      running    v0.18.1    linux/amd64, linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)
default                   docker                                                                                                                                  
 \_ default                \_ default                                                                                                       running    v0.16.0    linux/amd64, linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)
orbstack*                 docker                                                                                                                                  
 \_ orbstack               \_ orbstack                                                                                                      running    v0.16.0    linux/amd64, linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)

Configuration

N/A

Build logs

buildx create --debug --name charming_chaplygin --append --node buildx-arm64 --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 --driver-opt namespace=buildx --driver-opt requests.cpu=2 --driver-opt requests.memory=4G --driver-opt requests.ephemeral-storage=3Gi --driver-opt nodeselector=gha=true --driver-opt tolerations='key=build-agent,value=true' --driver-opt annotations=karpenter.sh/do-not-disrupt=true --platform linux/arm/v7
WARNING: new settings may not be used until builder is restarted
ERROR: failed to initialize builder charming_chaplygin (buildx-arm64): invalid driver option value for driver kubernetes
82388 v0.18.0 /Users/lawrenceaiello/.docker/cli-plugins/docker-buildx buildx create --debug --name charming_chaplygin --append --node buildx-arm64 --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 --driver-opt namespace=buildx --driver-opt requests.cpu=2 --driver-opt requests.memory=4G --driver-opt requests.ephemeral-storage=3Gi --driver-opt nodeselector=gha=true --driver-opt tolerations=key=build-agent,value=true --driver-opt annotations=karpenter.sh/do-not-disrupt=true --platform linux/arm/v7
github.com/docker/buildx/builder.Create
        github.com/docker/buildx/builder/builder.go:535
github.com/docker/buildx/commands.runCreate
        github.com/docker/buildx/commands/create.go:53
github.com/docker/buildx/commands.createCmd.func1
        github.com/docker/buildx/commands/create.go:99
github.com/docker/cli/cli-plugins/plugin.RunPlugin.func1.1.2
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:64
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:1117
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:1041
github.com/docker/cli/cli-plugins/plugin.RunPlugin
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:79
main.runPlugin
        github.com/docker/buildx/cmd/buildx/main.go:71
main.run
        github.com/docker/buildx/cmd/buildx/main.go:85
main.main
        github.com/docker/buildx/cmd/buildx/main.go:95
runtime.main
        runtime/proc.go:271
runtime.goexit
        runtime/asm_arm64.s:1222

Additional info

I was able to get it to work by escaping quotes around the toleration like so:

docker buildx create --debug --name charming_chaplygin --append --node buildx-arm64 --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 --driver-opt namespace=buildx --driver-opt requests.cpu=2 --driver-opt requests.memory=4G --driver-opt requests.ephemeral-storage=3Gi --driver-opt nodeselector=gha=true --driver-opt \"tolerations=key=build-agent,value=true\" --driver-opt annotations=karpenter.sh/do-not-disrupt=true --platform linux/arm/v7
WARNING: new settings may not be used until builder is restarted
charming_chaplygin

So at the very least I think the docs should be updated to reflect this. The better solution would be to fix it, this doesn't seem right.
@crazy-max
Copy link
Member

crazy-max commented Dec 16, 2024 •
edited
Loading

I think you need extra quotes:

docker buildx create --debug \
  --name charming_chaplygin \
  --append \
  --node buildx-arm64 \
  --driver-opt image=533267232274.dkr.ecr.us-east-1.amazonaws.com/dockerhub/moby/buildkit:v0.17.0 \
  --driver-opt namespace=buildx \
  --driver-opt requests.cpu=2 \
  --driver-opt requests.memory=4G \
  --driver-opt requests.ephemeral-storage=3Gi \
  --driver-opt nodeselector=gha=true \
  --driver-opt tolerations="key=build-agent,value=true" \
  --driver-opt annotations=karpenter.sh/do-not-disrupt=true \
  --platform linux/arm/v7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/driver/kubernetes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@crazy-max @aiell0