k8s terminal error on evicted pods

Summary

When you have evicted pod as follows:

 ❯ kubectl get pods --all-namespaces  | grep -i evicted                                                                                                                                          ⏎ 
gitlab          gitlab-runner-190353586-wnhc5                                            0/1       Evicted   0          5d 
gitlab          minio-966383792-kpp59                                                    0/1       Evicted   0          6d 
gitlab          runner-a1b569a9-project-119-concurrent-0cbfrf                            0/3       Evicted   0          23h 

The pod filtering does not properly get just the Running pods.

Steps to reproduce

Quite hard to reproduce manually, you should have the same pod in 2 states: running + evicted.

What is the current bug behavior?

Kubernetes terminal page 500.

What is the expected correct behavior?

Just to filter the running pod which should be ever equal to 1.

Relevant logs and/or screenshots

Completed 500 Internal Server Error in 182ms (ActiveRecord: 22.0ms)

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/gitlab/kubernetes.rb:47:in `add_terminal_auth'
  app/models/project_services/kubernetes_service.rb:114:in `block (2 levels) in terminals'
  app/models/project_services/kubernetes_service.rb:114:in `each'
  app/models/project_services/kubernetes_service.rb:114:in `block in terminals'
  app/models/concerns/reactive_caching.rb:65:in `block in with_reactive_cache'
  app/models/concerns/reactive_caching.rb:109:in `within_reactive_cache_lifetime'
  app/models/concerns/reactive_caching.rb:63:in `with_reactive_cache'
  app/models/project_services/kubernetes_service.rb:110:in `terminals'
  app/models/environment.rb:145:in `terminals'
  app/controllers/projects/environments_controller.rb:78:in `terminal_websocket_authorize'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

Output of checks

Results of GitLab environment info

System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.5.2
Bundler Version:1.14.6
Rake Version:	10.5.0
Redis Version:	2.8.4
Sidekiq Version:4.2.7

GitLab information
Version:	8.17.3
Revision:	f0c970e
Directory:	/home/git/gitlab
DB Adapter:	postgresql
URL:		https://gitlab.sparkfabrik.com
HTTP Clone URL:	https://gitlab.sparkfabrik.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.sparkfabrik.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	4.1.1
Repository storage paths:
- default: 	/home/git/data/repositories/
Hooks:		/home/git/gitlab-shell/hooks/
Git:		/usr/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
13/2 ... ok
14/3 ... ok
16/4 ... ok
14/5 ... ok
13/6 ... ok
19/7 ... ok
16/8 ... ok
16/9 ... ok
16/10 ... ok
11/12 ... ok
23/14 ... ok
23/15 ... ok
23/16 ... ok
16/17 ... ok
23/18 ... ok
24/19 ... ok
13/20 ... ok
13/21 ... ok
27/22 ... ok
27/23 ... ok
27/24 ... ok
13/27 ... ok
13/28 ... ok
29/37 ... ok
13/38 ... ok
31/40 ... ok
31/45 ... ok
11/46 ... ok
2/58 ... ok
6/85 ... ok
25/86 ... ok
11/87 ... ok
11/88 ... ok
31/89 ... ok
33/90 ... ok
34/91 ... ok
31/92 ... ok
16/93 ... ok
16/94 ... ok
16/95 ... ok
7/96 ... ok
7/97 ... ok
8/98 ... ok
5/99 ... ok
5/100 ... ok
20/101 ... ok
20/102 ... ok
8/103 ... ok
29/104 ... ok
29/105 ... ok
29/106 ... repository is empty
29/107 ... ok
29/108 ... repository is empty
29/109 ... repository is empty
20/110 ... ok
16/111 ... ok
20/112 ... ok
20/113 ... ok
42/114 ... ok
42/115 ... ok
46/116 ... ok
9/117 ... repository is empty
26/118 ... ok
75/119 ... ok
30/120 ... ok
49/121 ... ok
49/122 ... ok
16/123 ... ok
13/124 ... ok
13/125 ... ok
49/126 ... ok
49/127 ... ok
75/128 ... ok
49/129 ... ok
49/130 ... ok
49/131 ... ok
50/132 ... ok
51/133 ... ok
22/134 ... ok
49/135 ... ok
49/136 ... ok
53/137 ... ok
57/138 ... ok
55/139 ... ok
56/140 ... ok
59/141 ... ok
54/142 ... ok
54/143 ... ok
58/144 ... ok
58/145 ... ok
58/146 ... ok
55/147 ... ok
55/148 ... ok
24/150 ... ok
54/151 ... ok
54/152 ... ok
54/153 ... ok
54/154 ... ok
54/155 ... ok
58/156 ... repository is empty
61/157 ... ok
54/158 ... ok
54/159 ... ok
54/160 ... ok
54/161 ... ok
54/162 ... ok
54/163 ... ok
54/164 ... ok
54/165 ... ok
49/166 ... ok
59/167 ... repository is empty
8/168 ... ok
13/169 ... ok
46/170 ... ok
58/172 ... ok
60/173 ... ok
60/174 ... ok
60/175 ... ok
60/176 ... ok
60/177 ... ok
60/178 ... ok
60/179 ... ok
60/180 ... ok
22/181 ... ok
13/182 ... ok
67/183 ... ok
71/186 ... ok
73/188 ... ok
70/190 ... ok
58/191 ... ok
58/192 ... ok
58/193 ... ok
20/194 ... ok
58/195 ... ok
13/196 ... ok
49/197 ... ok
58/198 ... ok
13/199 ... ok
71/200 ... ok
58/201 ... ok
77/202 ... ok
58/203 ... ok
58/204 ... repository is empty
13/205 ... ok
16/206 ... repository is empty
58/207 ... ok
7/208 ... ok
58/210 ... ok
58/211 ... ok
50/213 ... ok
13/214 ... ok
75/216 ... ok
13/218 ... ok
50/219 ... ok
7/220 ... ok
50/221 ... ok
50/222 ... ok
75/223 ... ok
50/225 ... ok
75/226 ... ok
50/227 ... ok
58/228 ... ok
49/229 ... ok
49/230 ... ok
21/231 ... ok
58/232 ... ok
58/233 ... ok
7/234 ... ok
13/235 ... ok
58/236 ... ok
58/237 ... ok
50/238 ... ok
7/239 ... ok
75/240 ... ok
13/241 ... ok
58/242 ... ok
58/243 ... ok
89/244 ... ok
89/245 ... ok
89/246 ... ok
89/247 ... ok
89/248 ... repository is empty
58/249 ... ok
89/250 ... ok
89/251 ... ok
9/252 ... ok
49/253 ... ok
89/255 ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/git/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /home/git/data/uploads
  sudo find /home/git/data/uploads -type f -exec chmod 0644 {} \;
  sudo find /home/git/data/uploads -type d -not -path /home/git/data/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ...
13/2 ... yes
14/3 ... yes
16/4 ... yes
14/5 ... yes
13/6 ... yes
19/7 ... yes
16/8 ... yes
16/9 ... yes
16/10 ... yes
11/12 ... yes
23/14 ... yes
23/15 ... yes
23/16 ... yes
16/17 ... yes
23/18 ... yes
24/19 ... yes
13/20 ... yes
13/21 ... yes
27/22 ... yes
27/23 ... yes
27/24 ... yes
13/27 ... yes
13/28 ... yes
29/37 ... yes
13/38 ... yes
31/40 ... yes
31/45 ... yes
11/46 ... yes
2/58 ... yes
6/85 ... yes
25/86 ... yes
11/87 ... yes
11/88 ... yes
31/89 ... yes
33/90 ... yes
34/91 ... yes
31/92 ... yes
16/93 ... yes
16/94 ... yes
16/95 ... yes
7/96 ... yes
7/97 ... yes
8/98 ... yes
5/99 ... yes
5/100 ... yes
20/101 ... yes
20/102 ... yes
8/103 ... yes
29/104 ... yes
29/105 ... yes
29/106 ... yes
29/107 ... yes
29/108 ... yes
29/109 ... yes
20/110 ... yes
16/111 ... yes
20/112 ... yes
20/113 ... yes
42/114 ... yes
42/115 ... yes
46/116 ... yes
9/117 ... yes
26/118 ... yes
75/119 ... yes
30/120 ... yes
49/121 ... yes
49/122 ... yes
16/123 ... yes
13/124 ... yes
13/125 ... yes
49/126 ... yes
49/127 ... yes
75/128 ... yes
49/129 ... yes
49/130 ... yes
49/131 ... yes
50/132 ... yes
51/133 ... yes
22/134 ... yes
49/135 ... yes
49/136 ... yes
53/137 ... yes
57/138 ... yes
55/139 ... yes
56/140 ... yes
59/141 ... yes
54/142 ... yes
54/143 ... yes
58/144 ... yes
58/145 ... yes
58/146 ... yes
55/147 ... yes
55/148 ... yes
24/150 ... yes
54/151 ... yes
54/152 ... yes
54/153 ... yes
54/154 ... yes
54/155 ... yes
58/156 ... yes
61/157 ... yes
54/158 ... yes
54/159 ... yes
54/160 ... yes
54/161 ... yes
54/162 ... yes
54/163 ... yes
54/164 ... yes
54/165 ... yes
49/166 ... yes
59/167 ... yes
8/168 ... yes
13/169 ... yes
46/170 ... yes
58/172 ... yes
60/173 ... yes
60/174 ... yes
60/175 ... yes
60/176 ... yes
60/177 ... yes
60/178 ... yes
60/179 ... yes
60/180 ... yes
22/181 ... yes
13/182 ... yes
67/183 ... yes
71/186 ... yes
73/188 ... yes
70/190 ... yes
58/191 ... yes
58/192 ... yes
58/193 ... yes
20/194 ... yes
58/195 ... yes
13/196 ... yes
49/197 ... yes
58/198 ... yes
13/199 ... yes
71/200 ... yes
58/201 ... yes
77/202 ... yes
58/203 ... yes
58/204 ... yes
13/205 ... yes
16/206 ... yes
58/207 ... yes
7/208 ... yes
58/210 ... yes
58/211 ... yes
50/213 ... yes
13/214 ... yes
75/216 ... yes
13/218 ... yes
50/219 ... yes
7/220 ... yes
50/221 ... yes
50/222 ... yes
75/223 ... yes
50/225 ... yes
75/226 ... yes
50/227 ... yes
58/228 ... yes
49/229 ... yes
49/230 ... yes
21/231 ... yes
58/232 ... yes
58/233 ... yes
7/234 ... yes
13/235 ... yes
58/236 ... yes
58/237 ... yes
50/238 ... yes
7/239 ... yes
75/240 ... yes
13/241 ... yes
58/242 ... yes
58/243 ... yes
89/244 ... yes
89/245 ... yes
89/246 ... yes
89/247 ... yes
89/248 ... yes
58/249 ... yes
89/250 ... yes
89/251 ... yes
9/252 ... yes
49/253 ... yes
89/255 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.0)
Active users: 48

Checking GitLab ... Finished

Possible fixes

Just add here a new argument to filter for pod status:

lib/gitlab/kubernetes.rb:

    def filter_pods(pods, labels = {}, status = 'running')
      pods.select do |pod|
        metadata = pod.fetch("metadata", {})
        pod_labels = metadata.fetch("labels", nil)
        next unless pod_labels
        labels.all? { |k, v| pod_labels[k.to_s] == v }
      end
    end