Mix of pipeline and job level tags does not tag GHA job

[jmeickle-theaiinstitute]

Bug description

In Github Actions, when running this command in a job:

datadog-ci tag --level=pipeline --tags repro:dockerless &&
datadog-ci tag --level=job --tags jobrepro:dockerless-job

The Github Actions step output is:

Run datadog-ci tag --level=pipeline --tags repro:dockerless &&
  datadog-ci tag --level=pipeline --tags repro:dockerless &&
  datadog-ci tag --level=job --tags jobrepro:dockerless-job
  shell: bash --noprofile --norc -e -o pipefail {0}
  env:
   [REDACTED]
    DATADOG_API_KEY: ***
Run '/home/runner/k8s/index.js'
  shell: /home/runner/externals/node16/bin/node {0}
Tags sent
Tags sent

(That is, it runs twice successfully)

This is a case where we have one pipeline with two parallel jobs, both running these tagging commands. Using the example above, repro shows up as a tag on the pipeline but jobrepro does not show up as a tag on the job (in any run).

Describe what you expected

Either the datadog-ci tool should tag the data, or it should fail if this is an invalid usage

Steps to reproduce the issue

No response

Additional context

v2.33.1 of datadog-ci

Command

tag

[ManuelPalenzuelaDD]

Hi @jmeickle-theaiinstitute ! There are some limitations around adding custom tags to GitHub jobs.

We have some docs related to this. The TLDR is that if the job name is changed using the name property OR you're using a matrix strategy to define jobs you need to make expose a DD_GITHUB_JOB_NAME env variable pointing to the real job name.

Could this be whats going on here? 👀

[benjamin-cribb-jane]

datadog-ci tag --level job --tags pr_owner:<REDACTED>
  shell: /usr/bin/bash -e {0}
  env:
    PR_COMMENTS_ENABLED: false
    TAG: <REDACTED>
    DD_CIVISIBILITY_AGENTLESS_ENABLED: true
    DD_ENV: ci
    DD_SERVICE: fullstack
    DD_API_KEY: ***
    DD_GITHUB_JOB_NAME: Run e2e tests / Run E2E desktop on node 0
    DD_TAGS: pr_owner:<REDACTED>
    DATADOG_API_KEY: ***
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 59.2M  100 59.2M    0     0  [8]<REDACTED>
Tags sent

I've tried every possible combination of job name appearing anywhere in datadog or CICD. It's also silently failing to send tags to jobs which are not part of a matrix run. Both when tags are set as env variables, and when tags are attempted to be sent via datadog-ci, regardless of whether DD_GITHUB_JOB_NAME is provided or not provided.

[edit] Worth noting, my jobs are part of a reusable workflow triggered by a parent job

[ManuelPalenzuelaDD]

Hi @benjamin-cribb-jane , we recently had a bug that caused GitHub jobs' custom tags not to be added. We think that what might've happened is this:

1. The first issue was due to the missing DD_GITHUB_JOB_NAME variable.
2. We introduced a bug causing the GitHub jobs' custom tags behavior to break.
3. You tried using the DD_GITHUB_JOB_NAME variable and it didn't work (likely due to the bug introduced at step 2).
4. We fixed the bug introduced in step 2.

• Could you try doing this one more time with the DD_GITHUB_JOB_NAME variable?
• If you could also provide us an Org ID to look into this it'd be great.

Hopefully it works now! 🤞

[juan-fernandez]

hey did you try those steps @benjamin-cribb-jane ? 😄

[chrisforrette]

Hey there,

I'm experiencing similar issues to what's been described here.

To be specific, I'm not really seeing tags come through at the job level at all, though pipeline tags seem to work fine. I tried adding the DD_GITHUB_JOB_NAME environment variable described above as the job name is dynamic. This is what my (reduced) GitHub Actions YAML looks like:

jobs:
  build:
    name: Build ${{ inputs.service }}
    env:
      DD_GITHUB_JOB_NAME: Build ${{ inputs.service }}
    steps:
          - name: Datadog CI tags
             run: datadog-ci tag --level job --tags pod_name:${{ runner.name }}

In case the detail matters, technically the datadog-ci calls are being made from a GitHub Actions composite action. The output looks like this:

Run datadog-ci tag --level job --tags pod_name:gha-runner-large-dgflp-44nnq
  datadog-ci tag --level job --tags pod_name:gha-runner-large-dgflp-44nnq
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DD_GITHUB_JOB_NAME: Build admin-api
  
Tags sent

But if I go into Metrics Explorer, select the "CI Pipelines" dataset, set the "Level" to "Job", and open the "by" field I can see pod_name as an option, but if I select it nothing shows up in the graph. And if I filter with the value from the output above (e.g. pod_name:gha-runner-large-dgflp-44nnq) nothing shows up for any time interval.

In case it helps, my org ID is splice.

Please let me know if you have any suggestions, thank you!