Improve metrics

[tiithansen]

This PR makes following changes:

• Introduce new metrics gha_runner_job which can be used to link jobs to runner pods (metric is only exported while job is running).
• Replace high cardinality histograms with last duration gauges as jobs can run with long irregular intervals which makes rate functions hard to use.
• Add new duration gauge to show how long job sat in queue before being picked up.
• Fix name label to always contain the clean runnerScaleSetName (value used in GHA job runs-on property to select runner).
• Fix only export durations if both times used in duration calculations are set.
• Remove job_workflow_ref, runner_id and runner_name from duration metrics as they will cause a creation of a new metric/series with each run.

Example queries:

Memory usage per job:

label_replace(gha_runner_job{repository=~"$repository.*", job_name=~"$job.*"}, "pod", "$1", "pod_name", "(.*)") * on(pod) group_right(job_name) sum(container_memory_working_set_bytes{container!=""}) by (pod, container)

CPU usage per job:

label_replace(gha_runner_job{repository=~"$repository.*", job_name=~"$job.*"}, "pod", "$1", "pod_name", "(.*)") * on(pod) group_right(job_name) sum(rate(container_cpu_usage_seconds_total{container!=""}[1m])) by (pod, container)

CPU Throttling:

label_replace(gha_runner_job{repository=~"$repository.*", job_name=~"$job.*"}, "pod", "$1", "pod_name", "(.*)") * on(pod) group_right(job_name) sum(
    sum by (container,pod)
        (rate(container_cpu_cfs_throttled_periods_total{container!=""}[1m]))
 /
    sum by (container,pod)
        (rate(container_cpu_cfs_periods_total{container!=""}[1m]))
) by (pod, container)

[atsu85]

worth adding a comment (in addition to commit message) why Gague is used, while ideally Histogram seems better data type - might avoid a lot of WTFs and wasting time basically reverting this change ;)

[atsu85]

typo in commit description?

They cause a creation of new services with each job execution.

vs

They cause a creation of new series with each job execution.

Also worth mentioning cardinality explosion and OOMs