Exception iterating responses: Not a cmessage
Context
I am running buildgrid via kubernetees and the docker container here: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly.
I have a "stack" setup like so inside kubernetees:
BUILDGRID
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: buildgrid-stack
namespace: buildgrid
spec:
replicas: 1
selector:
matchLabels:
app: buildgrid-stack
serviceName: buildgrid-stack
template:
metadata:
labels:
app: buildgrid-stack
spec:
containers:
- args:
- /usr/local/bin/bgd
- server
- start
- /config/artifacts.conf
image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
name: buildgrid-stack
ports:
- containerPort: 50051
protocol: TCP
volumeMounts:
- mountPath: /config
name: config
- mountPath: /cas
name: cas
volumes:
- configMap:
name: config-storage
name: config
volumeClaimTemplates:
- metadata:
name: cas
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 12Gi
---
apiVersion: v1
data:
artifacts.conf: |
server:
- !channel
port: 50051
insecure-mode: true
instances:
- name: main
storages:
- !lru-storage &main-storage
size: 512MB
- !disk-storage &disk-storage
path: !expand-path $HOME/cas
services:
- !action-cache &main-action
storage: *main-storage
max-cached-refs: 256
allow-updates: true
- !execution
storage: *main-storage
action-cache: *main-action
- !cas
storage: *disk-storage
- !bytestream
storage: *main-storage
kind: ConfigMap
metadata:
name: config-storage
namespace: buildgrid
---
apiVersion: v1
kind: Service
metadata:
name: buildgrid-stack
namespace: buildgrid
spec:
ports:
- port: 50051
protocol: TCP
selector:
app: buildgrid-stack
type: LoadBalancer
I then have a worker deployed via kubernetees like so:
WORKER
apiVersion: apps/v1
kind: Deployment
metadata:
name: worker
namespace: buildgrid
spec:
replicas: 1
selector:
matchLabels:
app: worker
template:
metadata:
labels:
app: worker
spec:
containers:
- name: worker
image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
command: ["bgd","bot","--remote=http://buildgrid-stack:50051", "--parent=main", "host-tools"]
ports:
- containerPort: 50051
protocol: TCP
These are deploying with no errors and are staying active with 0 restarts from kubernetees.
I am then trying to test a minimal C application using this deployment in MiniKube, the port is exposed to localhost and point the Bazel build at the buildgrid stack at localhost:
bazel build :hello --spawn_strategy=remote --genrule_strategy=remote --remote_executor=localhost:50051 --remote_instance_name=main
The build begins and you can see Bazel has reached buildgrid from the server logs, however once the build is passed off to the worker, the build fails with this error:
INFO: Invocation ID: 399c623e-239f-4640-bc29-62b5a7c0d715
INFO: Analyzed target //:hello (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/adds68/git-repos/abseil-hello/bazel-hello/BUILD.bazel:15:1: C++ compilation of rule '//:hello' failed (Exit 34). Note: R
emote connection/protocol failed with: execution failed com.google.devtools.build.lib.remote.ExecutionStatusException: INTERNAL: Exce
ption iterating responses: Not a cmessage
Target //:hello failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 8.336s, Critical Path: 7.95s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
Checking the worker logs reports:
2019-05-28 16:18:17,918:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:18,426:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:18,933:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:19,439:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:19,944:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:20,450:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:20,965:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
2019-05-28 16:18:21,456:[ buildgrid.bot.tenant][ERROR]: Exception iterating responses: Not a cmessage
I am wondering if i have miss-configured something along this process, or i am seeing an genuine error with buildgrid runners?
Task Description
Investigate where this exception occurs in buildgrid.
To reproduce this you can use MiniKube and Kubectl with this branch: remote-apis-testing/remote-apis-testing!20 (merged)
minikube start
kubectl create -f kubernetees/buildgrid/
kubectl port-forward buildgrid-stack-0 50051:50051 -n buildgrid
bazel build :FOOBAR --spawn_strategy=remote --genrule_strategy=remote --remote_executor=localhost:50051 --remote_instance_name=main
Acceptance Criteria
When this exception can be mitigated.