Skip to content

[BB-1009] Update how we save configuration to use the prefix as key and a json …

Josue Balandrano Coronel requested to merge josuebc/BB-1009 into master

Update how we save configuration to use the prefix as key and a json object as value. Currently the configuration values are saved in multiple key/value pairs, e.g.:

[{
    "Key": "prefix/key1",
    "Value": "value1",
}, {
    "Key": "prefix/key2",
    "Value": "value2"
}
]

This PR will update the way the config values are saved to use the instance's prefix as key and the value will be a json object with all the configuration values. There are different methods implemented in the 'ConsulAgent' utility class to do basic CRUD opreations on the new formatted config objects.

Rationale

We are updating the way Consul saves instance's metadata but there are a lot of instances already running with a different metadata format. If we flip the switch and allow Ocim only handle one type of metadata format then we are going to have some down time while we update the other instance's metadata format. The code in this PR is backwards compatible and will only UPDATE old style metadata. For any new instance it will CREATE and UPDATE the new style of metadata. This works because consul KV store is built in a way that uses prefixed. So, if there's three KV pairs with the prefix ocmi/instances/121/ (e.g. ocim/instances/121/slug, ocim/instance/121/domains, ocim/instances/121/key) then Consul will only retrieve these values if kv.get is called with recurse=True if not consul will not return anything for kv.get("ocim/instances/121").

Testing instructions

  1. (Before pulling this PR) In a Django shell, create an instance:

instance = OpenEdXInstance.objects.create(
    name='Ironwood sandbox',
    sub_domain='ironwood',
    # The rest of the parameters are all optional:
    email='myname@opencraft.com',
    openedx_release='named-release/ironwood',
    configuration_version='named-release/ironwood',
    configuration_source_repo_url='https://github.com/edx/configuration.git',
    configuration_extra_settings='',
)
  1. Make sure the metadata was saved:
from instance.models.utils import ConsulAgent
ca = ConsulAgent(prefix='ocim/instances/<instasnce_id>')
ca._client.kv.get(ca.prefix, recurse=True)
  1. Pull this branch and update the created instance metadata
instsance.update_consul_metadata()
  1. Make sure nothing changed in the consul stored data .This is ensuring the code is backwards compatible.
from instance.models.utils import ConsulAgent
ca = ConsulAgent(prefix='ocim/instances/<instasnce_id>')
ca._client.kv.get(ca.prefix, recurse=True)
  1. Create a new instance and make sure the metadata was saved in the new format:

instance = OpenEdXInstance.objects.create(
    name='New sandbox',
    sub_domain='newsandbox',
    # The rest of the parameters are all optional:
    email='myname@opencraft.com',
    openedx_release='named-release/ironwood',
    configuration_version='named-release/ironwood',
    configuration_source_repo_url='https://github.com/edx/configuration.git',
    configuration_extra_settings='',
)
  1. Make sure the metadata was save in the new format:
from instance.models.utils import ConsulAgent
ca = ConsulAgent(prefix='ocim/instances/<instasnce_id>')
ca._client.kv.get(ca.prefix)

Merge request reports