Skip to content

Fix KeyError edge case in sql metrics

Marios Hadjimichael requested to merge marios/fix-empty-db-metrics-keyerror into master

While testing out BuildGrid with a sql backend, I realized that when starting with an empty database and the metrics enabled, we have an edge case in which a KeyError is thrown while trying to collect metrics due to all metric queries from the database returning null results.

2020-02-04 14:35:29,992:[buildgrid.server.persistence.sql.imp][ INFO][MainThread]: Creating SQL data store interface with: automigrate=[False], connection_timeout=[5] poll_interval=[1], kwargs=[{'pool_size': 5, 'max_overflow': 5, 'pool_timeout': 5}]
...
INFO][JobWatcher]: Starting job watcher thread
ExecutionService: Using SQL data store interface for `postgresql://bgd:***@mypostgres:1234/buildgrid` to store state
2020-02-04 14:35:30,030:[           buildgrid.server.instance][DEBUG][MainThread]: Setting up gRPC server with thread-limit=[300]
2020-02-04 14:35:30,032:[           buildgrid.server.instance][ INFO][MainThread]: Adding insecure connection on [[::]:50051]
2020-02-04 14:35:30,033:[           buildgrid.server.instance][ INFO][MainThread]: Server reflection is enabled for the following services:                                build.bazel.remote.execution.v2.Execution, google.longrunning.Operations, build.bazel.remote.execution.v2.ContentAddressableStorage
2020-02-04 14:35:30,035:[      buildgrid.server.bots.instance][ INFO][MainThread]: Starting BotSession reaper, bot_session_keepalive_timeout=[300].
2020-02-04 14:35:30,035:[      buildgrid.server.bots.instance][DEBUG][MainThread]: No more BotSessions to watch for expiry, waiting for new BotSessions.
2020-02-04 14:35:30,149:[buildgrid.server.persistence.sql.imp][ERROR][MainThread]: Error committing database session. Rolling back.
Traceback (most recent call last):
  File "/home/mhadjimichael/BuildGrid/buildgrid/env/lib/python3.7/site-packages/BuildGrid-0.0.2-py3.7.egg/buildgrid/server/persistence/sql/impl.py", line 185, in session
    yield session
  File "/home/mhadjimichael/BuildGrid/buildgrid/env/lib/python3.7/site-packages/BuildGrid-0.0.2-py3.7.egg/buildgrid/server/persistence/sql/impl.py", line 458, in get_metrics
    metrics[category] = category_cb(grouped_results[category])
KeyError: 'leases'

This MR addresses that.

Edited by Marios Hadjimichael

Merge request reports

Loading