Add operation status lock.
Description
grpc._server][ERROR]: Exception iterating responses: dictionary changed size during iteration
Traceback (most recent call last):
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/grpc/_server.py", line 452, in _take_response_from_response_iterator
return next(response_iterator), True
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/execution/service.py", line 108, in Execute
request.skip_cache_lookup)
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/execution/instance.py", line 87, in execute
skip_cache_lookup=skip_cache_lookup)
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/scheduler.py", line 231, in queue_job_action
self._update_job_operation_stage(job.name, operation_stage)
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/scheduler.py", line 523, in _update_job_operation_stage
self.__peer_message_queues)
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/job.py", line 375, in update_operation_stage
peer_message_queues=peer_message_queues)
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918T21505995/libexec/python3.6-buildgrid/lib/python3.6/site-packages/buildgrid/server/job.py", line 643, in _send_operations_updates
relevant_queues = {peer: mqs for peer, mqs in peer_message_queues.items()
File "/python3.6-buildgrid/0.0.2~20190918.1+b20190918 ... TRUNCATED
Lock to prevent concurrent access when updating operation status.
Edited by Arber X