TreeEntry: object size (1187469) is bigger than the maximum allowed size (1048576)
I first deployed the agent pointed at a config project with no manifest files. The agent config looks like:
gitops:
manifest_projects:
- id: appdat/platform/cluster-config
paths:
- glob: '/manifests/**/*.yaml'
The agent appeared to start successfully, indicating there were no manifests to sync:
{"level":"info","time":"2021-01-27T22:03:20.279Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"appdat/platform/cluster-config"}
{"level":"info","time":"2021-01-27T22:03:20.279Z","msg":"Start syncing cluster","mod_name":"gitops","project_id":"appdat/platform/cluster-config"}
{"level":"info","time":"2021-01-27T22:03:21.526Z","msg":"Throttling request took 1.199500701s, request: GET:https://10.12.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500\n"}
{"level":"info","time":"2021-01-27T22:03:31.726Z","msg":"Throttling request took 9.995709203s, request: GET:https://10.12.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500\n"}
{"level":"info","time":"2021-01-27T22:03:41.726Z","msg":"Throttling request took 9.793596801s, request: GET:https://10.12.0.1:443/apis/storage.cnrm.cloud.google.com/v1beta1/storagenotifications?limit=500\n"}
{"level":"info","time":"2021-01-27T22:03:48.541Z","msg":"Cluster successfully synced","mod_name":"gitops","project_id":"appdat/platform/cluster-config"}
{"level":"info","time":"2021-01-27T22:03:48.553Z","msg":"Syncing","mod_name":"gitops","project_id":"appdat/platform/cluster-config","skipHooks":true,"started":false}
{"level":"info","time":"2021-01-27T22:03:48.553Z","msg":"Tasks (dry-run)","mod_name":"gitops","project_id":"appdat/platform/cluster-config","tasks":"[]"}
{"level":"info","time":"2021-01-27T22:03:48.553Z","msg":"Updating operation state. phase: -> Succeeded, message: '' -> 'successfully synced (no more tasks)'","mod_name":"gitops","project_id":"appdat/platform/cluster-config"}
I then pushed manifests to the project under the manifests
directory while continuing to tail the pod logs and started seeing:
{"level":"error","time":"2021-01-27T22:05:33.005Z","msg":"GetObjectsToSynchronize.Recv failed","mod_name":"gitops","project_id":"appdat/platform/cluster-config","error":"rpc error: code = Unavailable desc = GitOps: failed to get objects to synchronize"}
The kas
server logs seem to indicate that the git repo size exceeds some maximum allowed size:
{"level":"info","time":"2021-01-27T22:17:48.591Z","msg":"GitOps: new commit","correlation_id":"01EX2YS59D4WGEDQ9RD1BT5YEG","agent_id":1,"project_id":"appdat/platform/cluster-config","commit_id":"e803b820156d4b76664d968b3423bddcf41d1c07"}
{"level":"error","time":"2021-01-27T22:17:48.616Z","msg":"GitOps: failed to get objects to synchronize","correlation_id":"01EX2YS59D4WGEDQ9RD1BT5YEG","agent_id":1,"project_id":"appdat/platform/cluster-config","commit_id":"e803b820156d4b76664d968b3423bddcf41d1c07","error":"TreeEntry.Recv: rpc error: code = FailedPrecondition desc = TreeEntry: object size (1187469) is bigger than the maximum allowed size (1048576)"}
Here's what my manifests directory looks like (note that manifests/certmanager/cert-manager/templates/crds.yaml
alone is approx 1.2MB):
$ du -h manifests
44K manifests/ingress/nginx-ingress/templates
48K manifests/ingress/nginx-ingress
52K manifests/ingress
60K manifests/prometheus/prometheus/templates
64K manifests/prometheus/prometheus
68K manifests/prometheus
12K manifests/elastic-stack/elastic-stack/charts/elasticsearch-curator/templates
16K manifests/elastic-stack/elastic-stack/charts/elasticsearch-curator
20K manifests/elastic-stack/elastic-stack/charts/elasticsearch/templates
24K manifests/elastic-stack/elastic-stack/charts/elasticsearch
24K manifests/elastic-stack/elastic-stack/charts/filebeat/templates
28K manifests/elastic-stack/elastic-stack/charts/filebeat
72K manifests/elastic-stack/elastic-stack/charts
76K manifests/elastic-stack/elastic-stack
80K manifests/elastic-stack
1.2M manifests/certmanager/cert-manager/templates
1.3M manifests/certmanager/cert-manager
1.3M manifests/certmanager
1.5M manifests
/cc @ash2k