[CI/CD Pipeline Error] error: pod, type/name or --filename must be specified

Summary

Failure on our broke our master pipeline. Creating this issue to collect more data when it occurs.

Examples

Error explanation

While trying to:

stdout, status = wait_for_dependencies
    fail stdout unless status.success?

the wait_for_dependenies tries to execute: cmd = full_command("/scripts/wait-for-deps")

full_command then tries to: "kubectl exec -it #{pod_name} -- env #{env_hash_to_str(env)} #{cmd}"

and finally, we have the error because pod_name returns something empty. One can confirm the error message by trying:

$ kubectl exec -it
error: pod, type/name or --filename must be specified

pod_name tries to get the name of the toolbox pod with:

    def pod_name
      filters = 'app=toolbox'

      @pod ||= find_pod_name(filters)
    end

    def find_pod_name(filters)
      if ENV['RELEASE_NAME']
        filters="#{filters},release=#{ENV['RELEASE_NAME']}"
      end

      `kubectl get pod -l #{filters} --field-selector=status.phase=Running -o jsonpath="{.items[0].metadata.name}"`
    end

This kubectl command got then an internal server error with not great feedback on what happened. It's unclear to me whether this internal server error came from our KAS CI/CD Tunnel mechanism, or from the Kubernetes API from GKE.

Relevant logs

Restoring a backup
Error from server (InternalError): an error on the server ("<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, ...6IiBmaWx") has prevented the request from succeeding (get pods)
error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:
	template was:
		{.items[0].metadata.name}
	object given to jsonpath engine was:
		map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}
Restored gitlab instance
    Project testproject1 should exist (FAILED - 1)
    Issue under testproject1 should exist (FAILED - 2)
    Test project repository should have Dockerfile (FAILED - 3)
    Should have at least 1 runner registered (FAILED - 4)
    Issue contains attachment in the description (FAILED - 5)
    Could pull image from registry (FAILED - 6)
  Backups
    Should be able to backup an identical tar (FAILED - 7)
Failures:
  1) Restoring a backup Restored gitlab instance Project testproject1 should exist
     Failure/Error: fail stdout unless status.success?
     RuntimeError:
       error: pod, type/name or --filename must be specified
     # ./spec/features/backups_spec.rb:7:in `block (2 levels) in <top (required)>'

The most important bits are:

Error from server (InternalError): an error on the server

and

RuntimeError:
       error: pod, type/name or --filename must be specified
     # ./spec/features/backups_spec.rb:7:in `block (2 levels) in <top (required)>'
Edited by João Alexandre Cunha