Skip to content

Issue 372 resource usage via console output

Chris Martin requested to merge cmart/exosphere:issue-372-resource-usage-via-console-output into master

Fixes #372 (closed) and #286 (closed).

Collecting server resource usage via the Nova console log and displaying it to the user.

  • Make script that logs resource use to console
    • Should work on CentOS 8 and Ubuntu 20 minimum. (Maybe also CentOS 7 and Ubuntu 18?)
    • If possible, avoid the need to install additional dependency packages
    • Should also be human-readable or no?
    • What to include in log entry:
      • Epoch
      • CPU load as a percentage across all cores (likely more user-friendly than load averages?)
      • Maybe also load averages?
      • RAM used/free (modulo https://www.linuxatemyram.com)
      • Space used/free on root disk, and possibly any mounted volumes!
  • Exosphere should deploy new instances with script and cron to run it
  • Exosphere code to GET recent console output and store it in model
    • Logic that determines polling frequency should handle these situations:
      • Server is logging resource usage to console
        • We should poll every minute, just the most recent few lines a couple seconds after we expect the next log entry (based on epoch in most recent last log entry)
        • If there is a lot of other output to the console for some reason, then get a bigger set of lines, maybe last 1000
      • Server was just created (<30 mins) and there is a lot of console output but logging is not happening yet, so we should keep getting the console log until we see it
      • Server was not just created (>30 mins) and isn't logging resource usage to console (not set up or been disabled), so we should stop polling it
    • When parsing console log output:
      • Try to decode each line as JSON
      • If we find a new resource usage log entry (whose epoch we don't already have stored), store it locally
      • Perhaps keep the last hour of history, discard the rest
  • Pretty graphs!
  • Figure out how to host the script.
    • Maybe we include it in the Exosphere repo
    • Maybe we "vendor" it in a separate repo on GitLab
    • Maybe we host it on try.exosphere.app
    • I ended up storing it as a snippet on GitLab, the same way we do the word list for the server password.

I wrote this script to make it work: $2015130

How to Test

Create a new server (CentOS 8 or Ubuntu 20, may also work with older distros). After a few minutes you should see pretty graphs.

Screenshots

image

Edited by Chris Martin

Merge request reports