Pod label name cannot contain special characters with Kubernetes executor

Summary

We have variable CI_JOB_NAME and its value lets say is "1/5-owasp-security-scan".

When the executor creates the pod, it ends up generating a pod label from the variable CI_JOB_NAME with the configuration below:

[runners.kubernetes.pod_labels ] 
runner = "kubernetes-medium" 
project = "$CI_PROJECT_NAME" 
job = "$CI_JOB_NAME" 
team = "$CI_PROJECT_NAMESPACE"

When the executor creates the pod, it ends up generating a pod label: job: "1/5-owasp-security-scan". Labels cannot contain any of the special separators valid for gitlab (in this case /) to group the job, so the pod fails to start.

Perhaps we could sanitize pod labels before they are applied in the Kubernetes executor: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/executors/kubernetes/executor_kubernetes.go#L489

Something similar to:

labels[k] = regexp.MustCompile(`[^-a-z0-9A-Z_.]+`).ReplaceAll(s.Build.Variables.ExpandValue(v), "_")

ZD: https://gitlab.zendesk.com/agent/tickets/126601