Bulk API fails with Job not found error

[chandrams]

Describe the bug

Invoking bulk service with parallel requests with thanos datasource and multiple kruize replicas failed with job not found error

How to reproduce it

• Get the bulk scale test from this PR - Included kruize local scalability test using bulk API #1396
• Connect to the OCP cluster
• Update kruize replicas to 3
• Execute the below cmd
  ./bulk_scale_test.sh -i quay.io/kruize/autotune_operator:0.2 -w 5 -r /tmp/results

Expected behavior
Recommendations should be generated

Relevant logs
Kruize pod log

2024-12-0206:22:05.475 INFO [main][EMExecutorService.java(67)]-Starting stage processors
Waiting for experiments ... 
2024-12-0206:22:45.095 INFO [qtp1558103808-43][BulkService.java(74)]-Job ID: e751d01d-fcfb-4a1c-938d-97bae5bc232f
2024-12-0206:22:45.095 INFO [qtp1558103808-46][BulkService.java(74)]-Job ID: d0ec7b4b-d6a6-4004-bb8e-ce410469b94c
2024-12-0206:22:45.113 INFO [qtp1558103808-49][BulkService.java(74)]-Job ID: a66ece65-e62e-441d-8816-9518a59c1653
2024-12-0206:22:45.144 ERROR [qtp1558103808-43][KruizeErrorHandler.java(71)]-{"message":"Job not found","httpcode":404,"documentationLink":"","status":"ERROR"}
2024-12-0206:22:45.144 ERROR [qtp1558103808-46][KruizeErrorHandler.java(71)]-{"message":"Job not found","httpcode":404,"documentationLink":"","status":"ERROR"}
2024-12-0206:22:45.144 ERROR [qtp1558103808-49][KruizeErrorHandler.java(71)]-{"message":"Job not found","httpcode":404,"documentationLink":"","status":"ERROR"}
2024-12-0206:22:55.144 INFO [qtp1558103808-45][BulkService.java(74)]-Job ID: f0dc4f74-c128-4999-8df3-cfd73771f534
2024-12-0206:22:55.145 ERROR [qtp1558103808-45][KruizeErrorHandler.java(71)]-{"message":"Job not found","httpcode":404,"documentationLink":"","status":"ERROR"}

Environment:

• Kubernetes Cluster - openshift