Skip to content

Allow setting additional monitoring emails per instance (OC-2496)

Boros Gábor requested to merge newrelic-additional-monitoring into master

Created by: bradenmacdonald

This PR introduces a new per-instance setting, additional_monitoring_emails, which can be used to define extra emails (in addition to those in settings.ADMINS) who should receive email alerts from New Relic for that instance. The setting can be set via the shell or the django admin, and it appears in both the API and the UI.

Test Instructions:

  1. Log in to New Relic, select the OpenCraft.Hosting account, go to Account Settings, API Keys, and create/note your "Admin user's API key" (we do not need the "REST API key" even though we are using a REST API).
  2. In your OC IM devstack .env file, configure the following settings (substitute the real values): 
    NEWRELIC_ADMIN_USER_API_KEY='your_admin_api_key_here'
ADMINS = '[["Your Name", "your_email@opencraft.com"]]'
  3. On your OC IM devstack, have at least one instance that is configured but offline. It does not need to have any associated online VMs, so there's no need to go through the provisioning process.
  4. Go the the django admin, find that instance, and set its "Additional monitoring emails" to "your_email+somethingelse@opencraft.com"
  5. In the UI, ensure that additional email is shown on the instance's "Settings" tab (you'll need to force your browser to ignore its cache of angular HTML templates)
  6. On your devstack, run make shell
  7. In the shell, find the instance by its ID, e.g. inst = InstanceReference.objects.get(pk=50).instance
  8. Enable monitoring with inst.enable_monitoring()
  9. Within a few minutes, you should get three emails from New Relic indicating that the instance's three URLs are offline. Verify that each email was sent to both your_email@opencraft.com (from ADMINS) and your_email+somethingelse@opencraft.com (from "Additional monitoring emails")
  10. Run inst.disable_monitoring() to delete the monitors

Other Notes: I didn't have time during this task, but at some point we should upgrade to the New Relic Synthetics API v3 instead of v1, which is deprecated. The parts of the API that we use appear to be unchanged, but more thorough testing would be required to say for sure.

FYI: API documentation: https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/synthetics-api/synthetics-rest-api-version-1

Merge request reports