Remote Execution tests produce too much logs

Summary

The Remote Execution job that we have in CI produces large amounts of logs on each run. Take https://gitlab.com/BuildStream/buildstream/-/jobs/335573780 as an example. This job isn't particularly special, and endded up producing 32 MBs of logs. I think that's a bit excessive, and makes it harder to find useful information from the logs.

Also, the GitLab CI gives up showing the logs at this size, and only displays debug logs from the bots. So, the entire screen ends up being filled with the folloing pairs of log lines:

...
 2019-10-28 23:24:17,798:[      buildgrid.server.bots.instance][DEBUG]: Sending session update, name=[/975ca549-f7f2-45ef-adec-ef6f519569b7], for bot=[.20b87ff6683b.1], leases=[]
controller_1  | 2019-10-28 23:24:17,802:[       buildgrid.server.bots.service][DEBUG]: UpdateBotSession request from [ipv4:172.19.0.4:33572]
controller_1  | 2019-10-28 23:24:17,807:[      buildgrid.server.bots.instance][DEBUG]: Sending session update, name=[/975ca549-f7f2-45ef-adec-ef6f519569b7], for bot=[.20b87ff6683b.1], leases=[]
...

Steps to reproduce

Open logs for any remote execution job on CI.

What is the current bug behavior?

Large amount of logs.

What is the expected correct behavior?

More managable logs on CI output.

Possible fixes

Consider tuning the log level for stuff that gets printed on stdout/stderr. If debug logs are really needed, maybe put them in a separate file.

added Remote Execution Tests labels

mentioned in merge request !1671 (merged)

assigned to @traveltissues

afaict there's no native option for bgd bot and bgd server to log to a file, only verbosity control, we could try to direct it

mentioned in commit b1ef7072

mentioned in commit 9ae7b679

closed via commit 9ae7b679

closed via commit 2bb3dc15

closed via merge request !1677 (merged)

mentioned in commit 2bb3dc15