Ugly unhandled exception when cache server restarted

Summary

Over on freedesktop-sdk I recently had a pipeline fail when @jjardon restarted the cache server, with an ugly stack trace.

Steps to reproduce

Restart a server while an artifact is being pulled.

What is the current bug behavior?

The build/pull will fail with an ugly stack trace

What is the expected correct behavior?

• A polite error message
• (better) Do not fail at all: continue the build even if the cache server fails for some reason

Relevant logs and/or screenshots

[36m[2m[[0m[33m00[0m[36m[2m:[0m[33m00[0m[36m[2m:[0m[33m20[0m[36m[2m][[0m[31m[2m29a0ed0c[0m[36m[2m][[0m[33m pull[0m[36m[2m:[0m[33mdesktop/xorg-lib-xxf86vm.bst  [0m[36m[2m] [0m[31m[1m[2mBUG    [0m[36m[2m [0mPull

    [31m[1m[2m    An unhandled exception occured:
    
        Traceback (most recent call last):
          File "/root/.local/lib/python3.6/site-packages/buildstream/_scheduler/jobs/job.py", line 398, in _child_action
            result = self.child_process()
          File "/root/.local/lib/python3.6/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 94, in child_process
            return self._action_cb(self._element)
          File "/root/.local/lib/python3.6/site-packages/buildstream/_scheduler/queues/pullqueue.py", line 36, in process
            return element._pull()
          File "/root/.local/lib/python3.6/site-packages/buildstream/element.py", line 1616, in _pull
            pulled = self._pull_strong(progress=progress)
          File "/root/.local/lib/python3.6/site-packages/buildstream/element.py", line 1580, in _pull_strong
            if not self.__artifacts.pull(self, key, progress=progress):
          File "/root/.local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 226, in pull
            response = remote.ref_storage.GetReference(request)
          File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 514, in __call__
            return _end_unary_response_blocking(state, call, False, None)
          File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 448, in _end_unary_response_blocking
            raise _Rendezvous(state, None, None, deadline)
        grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        	status = StatusCode.UNAVAILABLE
        	details = "Connect Failed"
        	debug_error_string = "{"created":"@1533226823.557230502","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2673,"referenced_errors":[{"created":"@1533226823.557203181","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":210,"referenced_errors":[{"created":"@1533226823.556671571","description":"Connect Failed","file":"src/core/ext/filters/client_channel/subchannel.cc","file_line":641,"grpc_status":14,"referenced_errors":[{"created":"@1533226823.556559218","description":"Failed to connect to remote host: FD Shutdown","file":"src/core/lib/iomgr/lockfree_event.cc","file_line":190,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1533226823.556518428","description":"connect() timed out","file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:209.97.137.202:11002"}]}]}]}"
        >[0m

[33m[1mPipeline Summary
[0m[36m[2m  Total:       [0m[33m330[0m
[36m[2m  Session:     [0m[33m330[0m
[36m[2m  Pull Queue:  [0m[33mprocessed [0m[32m78[0m[36m[2m, [0m[33mskipped [0m[33m140[0m[36m[2m, [0m[33mfailed [0m[31m[2m37[0m 
[36m[2m  Fetch Queue: [0m[33mprocessed [0m[32m81[0m[36m[2m, [0m[33mskipped [0m[33m73[0m[36m[2m, [0m [33mfailed [0m[31m[2m0[0m  
[36m[2m  Build Queue: [0m[33mprocessed [0m[32m0[0m[36m[2m, [0m [33mskipped [0m[33m50[0m[36m[2m, [0m [33mfailed [0m[31m[2m0[0m  
[36m[2m  Push Queue:  [0m[33mprocessed [0m[32m42[0m[36m[2m, [0m[33mskipped [0m[33m8[0m[36m[2m, [0m  [33mfailed [0m[31m[2m0[0m  


section_end:1533227474:build_script
[0Ksection_start:1533227474:after_script
[0Ksection_end:1533227477:after_script
[0Ksection_start:1533227477:upload_artifacts
[0K[32;1mUploading artifacts...[0;m
/builds/freedesktop-sdk/freedesktop-sdk/cache/buildstream/logs: found 942 matching files[0;m 
Uploading artifacts to coordinator... ok          [0;m  id[0;m=86395870 responseStatus[0;m=201 Created token[0;m=bDX_5r6K
section_end:1533227482:upload_artifacts
[0K[31;1mERROR: Job failed: exit code 1
[0;m

That stack trace but several times over.

---