aegir issueshttps://gitlab.com/aegir/aegir/-/issues2021-10-24T20:19:17Zhttps://gitlab.com/aegir/aegir/-/issues/79Switch to DDEV for local dev2021-10-24T20:19:17ZChristopher GervaisSwitch to DDEV for local devImmediate tasks:
* [x] Add DDEV commands to:
* [x] Update/restart dispatcherd worker
* [x] Update/restart relayd worker
* [x] Add DDEV docker compose config
Assuming that the fall-out from [DDEV-live being discontinued](https:/...Immediate tasks:
* [x] Add DDEV commands to:
* [x] Update/restart dispatcherd worker
* [x] Update/restart relayd worker
* [x] Add DDEV docker compose config
Assuming that the fall-out from [DDEV-live being discontinued](https://www.ddev.com/notice/) is reasonable, we should switch to DDEV-local. We need to take more direct control:
* networking between containers
* mounted volumes
This is specifically because we're trying to use containers to act as micro-VMs, and so we need them to behave more like it.pre-Alpha0 ReleaseChristopher GervaisChristopher Gervaishttps://gitlab.com/aegir/aegir/-/issues/71Update Operation logging to use relayd mechanism2021-10-25T04:04:16ZDerek LaventureUpdate Operation logging to use relayd mechanismComing from #64, once we have aegir:input supporting append mode, we can replace the current Operation logging mechanism (that posts to an http endpoint) with the new `relayd`/`aegir:input` mechanism. Once that's done, we can strip out t...Coming from #64, once we have aegir:input supporting append mode, we can replace the current Operation logging mechanism (that posts to an http endpoint) with the new `relayd`/`aegir:input` mechanism. Once that's done, we can strip out the HTTP endpoints, routes, etc. currently used for this on the Drupal side.
TODO:
* [x] replace HTTP POST in `dispatcherd` with a `relayd` task to call `aegir:log` in append mode
* [x] Handle post-exit-code functionality in a similar fashion
* [x] `drupal aegir:exit-code` command
* [x] `relayd` post_exitcode task
* [x] `dispatcherd` relay_exitcode task
* [x] remove existing HTTP endpoints on Drupal frontend
* [x] Remove HTTP-based log route (`aegir_operation.receive.log.output`) and controller
* [x] Remove HTTP-based exitcode route (`aegir_operation.receive.log.exitcode`) and controller
* [x] add Operation logging-specific wrappers in `dispatcherd` and `relayd`
* [x] update `dispatcherd` to replace post to HTTP log route with new `aegir_log` functionality on `relayd`
* [ ] ~~Remove 'sequence' parameter/base-field~~ see #89pre-Alpha0 Releasehttps://gitlab.com/aegir/aegir/-/issues/68Implement Ansible action module for logging2021-11-18T16:55:23ZDerek LaventureImplement Ansible action module for loggingComing from #64 and #67, we plan to create a new Ansible action module to allow backend tasks to log their operations directly back to the frontend from Ansible roles themselves. This will likely replace the existing `warn` module, to pr...Coming from #64 and #67, we plan to create a new Ansible action module to allow backend tasks to log their operations directly back to the frontend from Ansible roles themselves. This will likely replace the existing `warn` module, to provide a way for backend roles to inject their own output into the logs `dispatcherd` is picking up.
TODO:
* [x] Un-escape log lines in `aegir:log`
* [x] Update `example_print_debug_data` ansible "task" role to use `aegir_log`:
* [x] emit basic (fixed) string log message
* [x] emit (variable) field data as a log message
* [x] copy `aegir_input.py` to `aegir_log.py` and adjust as needed
* [x] rough in `aegir_log.py`
* [x] write tests for:
* [x] drupal aegir:log
* [x] dispatcherd relay_aegir_log task (not really viable stand-alone. done via tests of `aegir:log` command)
* [x] relayd aegir_log task (not really viable stand-alone. done via tests of `aegir:log` command)
* [x] Add a test step to look for output from the `aegir_log` action module emitted by `example_print_debug_data` Ansible "task" role in Example Site test Scenario (`profile/modules/examples/example_site/features/example_site.feature`) to illustrate the default Operation running/logging behaviour. (followup in #58)
* [x] create `aegir:log` console command to receive log data: sequence (for now), timestamp, and data
* [x] create `log_command_test` fixtures module to provide a simple Operation (no tasks needed, the base fields are all we need to log data)
* [x] add log_command_test Feature to test the `aegir:log` command (using `aegir:input` as a model)
* [x] implement Behat "Given :type operations:" step definition to instantiate a fixture Operation for the test
* [x] update docs:
* [x] to reflect the new logging and input architecture via `relayd`
* [x] how to use the `aegir_log` action module in a "task" rolepre-Alpha0 ReleaseDerek LaventureChristopher GervaisDerek Laventurehttps://gitlab.com/aegir/aegir/-/issues/66Architecture docs updates2021-11-26T05:45:19ZDerek LaventureArchitecture docs updatesIn #64 and related tasks, we have changed the architecture of the system fairly significantly. We need to update the docs to reflect these changes:
* [x] translate the notes on this issue (and esp [this comment](https://gitlab.com/aegir...In #64 and related tasks, we have changed the architecture of the system fairly significantly. We need to update the docs to reflect these changes:
* [x] translate the notes on this issue (and esp [this comment](https://gitlab.com/aegir/aegir/-/issues/64#note_486779869) into (draft) docs to detail this aspect of the architecture better.
* [x] update Documentation to reflect these architecture changes
* [x] https://docs.aegir.hosting/dev/architecture/
* [x] https://docs.aegir.hosting/dev/architecture/sequence/
* [x] https://docs.aegir.hosting/dev/roadmap/
* [x] modify/revise the sequence diagram to reflect this in the docs (https://docs.aegir.hosting/dev/architecture/sequence/)
* [x] add an architecture diagram specific to the RabbitMQ/Celery + relayd/dispatcherd + `drupal aegir` console commands
* [x] add a Glossary:
* message queue vs. task queue
* exchange, routing_key, etc. in Celery?pre-Alpha0 Releasehttps://gitlab.com/aegir/aegir/-/issues/59Setup packer container image scripts2021-11-20T04:18:52ZDerek LaventureSetup packer container image scriptsImmediate tasks:
* [x] Clean up Ansible cruft
* [x] configure Supervisor (on dispatcherd) for:
* [x] RabbitMQ
* [x] Redis
* [x] dispatcherd worker
* [x] Add `start.sh` to run Supervisor in non-daemon mode (see [here](https://...Immediate tasks:
* [x] Clean up Ansible cruft
* [x] configure Supervisor (on dispatcherd) for:
* [x] RabbitMQ
* [x] Redis
* [x] dispatcherd worker
* [x] Add `start.sh` to run Supervisor in non-daemon mode (see [here](https://github.com/drud/ddev/blob/master/containers/ddev-webserver/ddev-webserver-prod-scripts/start.sh#L93))
* [x] Split up `aegir.relayd/tasks/main.yml` to allow for easier re-use
* [x] (maybe?) Add relayd-base image (just dependencies)
* [x] Configure Supervisor (on relayd) for:
* [x] relayd worker
* [x] Add `docker login` target (and maybe others)
* [x] Add tests:
* [x] `test-dispatcherd.yml`
* [x] `test-relayd.yml`
* [x] Clean up Packer cruft
* [x] Move Packer scripts under `build`
The immediate goal here is to shift the install and setup out of Lando/DDEV of the stack of services software we rely on to get Aegir going in a Development environment. Instead, we should bake as much as possible into the container, so the dev environment simply starts the container and kicks off a "start aegir" type command (initscript, daemon-mode systemd service via systemd-run, or simple shell script in /usr/local/bin for now).
To some extent we want to model the way that this will actually work in a production container-driven infrastructure, but the details of that are secondary to the current need to bootstrap a development environment with a working and reasonably thorough test suite, so we can confidently refactor things at all levels, including the service layer.
Initially the goal is to mirror what we're doing in Lando currently, in conjunction with #79's effort to move to DDEV, and mapping our lando setup scripts and Ansible playbooks for relayd and dispatcherd into a Packer-driven mechanism in order to speed up spinning up a dev environment.
Layers of container images (goal state):
* `10-focal.json` builds a root image with the relevant OS
* `20-base.json` builds in some extra "base" packages like `make`, `sudo`, `screen`, `strace`, `htop`, `vim`, etc. ("slightly opinionated Ubuntu")
* `30-python.json` builds in the python basics, needed for all three "top-level" images
* `40-relayd.json` builds a "frontend" aegir image, with PHP and the Drupal stack, ready to install Aegir's frontend site.
* only needs celery php library, and enough python to run `relayd.py` (and not rabbitmq)
* `40-dispatcherd.json` builds a "backend" aegir image
* needs fully rabbitmq + redis + celery setup
* needs ansible
* might need `php-cli`?
* from DDEV perspective, goal is to install up to the point that container "entrypoint" command can be our `celery multi start` command
* `40-minion.json` builds a bare-bones image to serves as the basis for both:
* other "flavours" of minion (ie. with a LAMP stack pre-configured, for Aegir to deploy Drupal sites on, or a Ruby stack on which to deploy Rails apps)
* a "clean slate" image for Aegir Server & Service entity Operations to provision via the frontend Aegir UI
* **KEY** to a minion is that it has a mechanism (initially just baked into the Packer build scripts) to generate and provision SSH keys so that the `dispatcherd` container can SSH to its minions immediately upon spinning up
* over the medium term, we'll want a more secure mechanism to inject these as the container is spun up, and for updating/removing keypairs from an environment
Coming from #75, let's make sure to update:
* [ ] ~~PHP (php8?)~~ see #84
* [x] MySQL (maria)
* [x] Ansible (latest)
* [x] s/bionic/focal/pre-Alpha0 ReleaseDerek LaventureDerek Laventurehttps://gitlab.com/aegir/aegir/-/issues/55Add regular user / site builder to roadmap2021-11-26T05:45:26ZColan SchwartzAdd regular user / site builder to roadmapIn the https://docs.aegir.hosting/dev/roadmap/, we're neglecting to mention users of the site that don't care about sysadmin things; they just want to get stuff deployed. Let's add this role.
Thanks @jonpugh for raising this.
CC @sens...In the https://docs.aegir.hosting/dev/roadmap/, we're neglecting to mention users of the site that don't care about sysadmin things; they just want to get stuff deployed. Let's add this role.
Thanks @jonpugh for raising this.
CC @sensespidey @ergonlogicpre-Alpha0 Releasehttps://gitlab.com/aegir/aegir/-/issues/81Move README details into Getting Started docs2021-07-09T19:43:56ZDerek LaventureMove README details into Getting Started docsThis page: https://docs.aegir.hosting/getting-started/ should have the contents of the [README](README.md)This page: https://docs.aegir.hosting/getting-started/ should have the contents of the [README](README.md)Alpha0 ReleaseSeonaid Leeseonaid@consensus.enterprisesSeonaid Leeseonaid@consensus.enterprises2021-07-09https://gitlab.com/aegir/aegir/-/issues/172Deploy a Project Environment (VM Deployment Target, Ansible backend)2023-11-01T19:33:15ZDerek LaventureDeploy a Project Environment (VM Deployment Target, Ansible backend)> As an Aegir User
>
> In order to host a Project Environment to a VM Deployment Target using an Ansible backend
>
> I need to deploy a Project Environment
Notes:
* the Project bundle determines what Operations and Tasks are instantiat...> As an Aegir User
>
> In order to host a Project Environment to a VM Deployment Target using an Ansible backend
>
> I need to deploy a Project Environment
Notes:
* the Project bundle determines what Operations and Tasks are instantiated, and therefore what kind of Deployment Target and Backend task runners are available/used.Release 2 - Stateful Project support (Drupal)https://gitlab.com/aegir/aegir/-/issues/171Create a Project Environment entity (VM Deployment Target, Ansible backend)2023-11-01T19:27:52ZDerek LaventureCreate a Project Environment entity (VM Deployment Target, Ansible backend)> As an Aegir User
>
> In order to host a Project Environment on a VM Deployment Target with Ansible backend
>
> I need to create a Project Environment entity
Notes:
* Associated with a Project (entityreference), so we inherit codebase...> As an Aegir User
>
> In order to host a Project Environment on a VM Deployment Target with Ansible backend
>
> I need to create a Project Environment entity
Notes:
* Associated with a Project (entityreference), so we inherit codebase, etc.
* Associate my project with a *Deployment Target*
* URL may vary
* Branch may vary
* Deployment Target may varyRelease 2 - Stateful Project support (Drupal)https://gitlab.com/aegir/aegir/-/issues/170Verify a Drupal Project2023-11-01T19:28:41ZDerek LaventureVerify a Drupal Project> As an Aegir user
>
> In order to initialize a Drupal project
>
> I need to verify access to the codebase
Notes:
* Verify access to the codebase
* This is probably just a shell backend operation that runs on the dispatcher to confirm...> As an Aegir user
>
> In order to initialize a Drupal project
>
> I need to verify access to the codebase
Notes:
* Verify access to the codebase
* This is probably just a shell backend operation that runs on the dispatcher to confirm the git repo details we have allow us to clone it successfully.Release 2 - Stateful Project support (Drupal)https://gitlab.com/aegir/aegir/-/issues/169Create a Drupal Project entity2023-11-01T19:27:29ZDerek LaventureCreate a Drupal Project entity> As an Aegir user
>
> In order to initialize a Drupal Project
>
> I need to create a Drupal Project entity
* Create a *Project* to represent my Drupal site
* Option: point to an existing Drupal codebase (default)
* Option: init...> As an Aegir user
>
> In order to initialize a Drupal Project
>
> I need to create a Drupal Project entity
* Create a *Project* to represent my Drupal site
* Option: point to an existing Drupal codebase (default)
* Option: initialize a Drupal codebase (`composer create-project`)
* Option: clone a pre-baked Drupal distribution
* After the entity is created, we should immediately trigger a Verify Project operationRelease 2 - Stateful Project support (Drupal)https://gitlab.com/aegir/aegir/-/issues/168Verify access to a K8S Deployment Target2023-11-07T15:28:35ZDerek LaventureVerify access to a K8S Deployment Target> As an Aegir manager,
>
> In order to make a *K8S Deployment Target* available
>
> I need to run a verify operation on a K8S Deployment Target
* This is an Operation (button) whose tasks will need to run remotely on the K8S cluster
...> As an Aegir manager,
>
> In order to make a *K8S Deployment Target* available
>
> I need to run a verify operation on a K8S Deployment Target
* This is an Operation (button) whose tasks will need to run remotely on the K8S cluster
* Marshal the variables from the Operation's tasks
* Determine the backend for each task
* Dispatch each task to the appropriate backend
* Tasks:
* Confirm credentials (`kubectl get all` to the server)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/167Create a K8S Deployment Target2023-11-07T15:24:15ZDerek LaventureCreate a K8S Deployment Target> As an Aegir manager,
>
> In order to make a *K8S Deployment Target* available
>
> I need to create a K8S Deployment Target
Architecture notes:
* This is creating an entity using the "K8S" bundle of the Deployment Target entity type
* ...> As an Aegir manager,
>
> In order to make a *K8S Deployment Target* available
>
> I need to create a K8S Deployment Target
Architecture notes:
* This is creating an entity using the "K8S" bundle of the Deployment Target entity type
* Fields:
* Target name (host)
* Routeable address (hostname or IP)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/166Deploy a static html Project (K8S Deployment Target)2023-11-07T15:28:40ZDerek LaventureDeploy a static html Project (K8S Deployment Target)> As an Aegir user,
>
> In order to deploy my static html project to a K8S Deployment Target
>
> I need to run a deploy operation on my static html Project
* this is an Operation (button) whose tasks will need to run remotely on the K...> As an Aegir user,
>
> In order to deploy my static html project to a K8S Deployment Target
>
> I need to run a deploy operation on my static html Project
* this is an Operation (button) whose tasks will need to run remotely on the K8S cluster
* Marshal the variables from the Operation's tasks
* Determine the backend for each task
* Dispatch each task to the appropriate backend
* Tasks:
* Build a Docker image (from a template?)
* Based on basic Nginx container image
* Clone the git repo (from the "static html Project")
* Create a Deployment resource for K8S
* ClusterIP service
* Ingress service
* Deployment (for now?)
* `kubectl apply`Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/165Create a static html Project (K8S Deployment Target)2023-11-07T15:24:28ZDerek LaventureCreate a static html Project (K8S Deployment Target)> As an Aegir user,
>
> In order to deploy a **static html site** to a **K8S Deployment Target**
>
> I need to create a static html Project
Architecture notes:
* This is creating an entity (re)using the "static html" bundle of the Proje...> As an Aegir user,
>
> In order to deploy a **static html site** to a **K8S Deployment Target**
>
> I need to create a static html Project
Architecture notes:
* This is creating an entity (re)using the "static html" bundle of the Project entity type
* Fields:
* hostname (must be unique)
* Git repo
* public url or embedded deploy token
* default branch (this is essentially a "singleton" project)
* Select a Deployment Target (K8S)
* Output: Print instruction point the hostname towards the site (deployment target IP address or hostname)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/164Verify access to a VM Deployment Target2023-11-07T15:28:07ZChristopher GervaisVerify access to a VM Deployment Target> As an Aegir manager,
>
> In order to make a *VM Deployment Target* available
>
> I need to run a verify operation on a VM Deployment Target
* This is an Operation (button) whose tasks will need to run remotely on the VM
* Marshal ...> As an Aegir manager,
>
> In order to make a *VM Deployment Target* available
>
> I need to run a verify operation on a VM Deployment Target
* This is an Operation (button) whose tasks will need to run remotely on the VM
* Marshal the variables from the Operation's tasks
* Determine the backend for each task
* Dispatch each task to the appropriate backend
* Tasks:
* Confirm credentials (ssh to the server)
* Confirm presence and write-access to root DocumentRoot (sudo touch /var/aegir/...)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/163Create a VM Deployment Target2023-12-09T23:01:38ZChristopher GervaisCreate a VM Deployment Target> As an Aegir manager,
>
> In order to make a *VM Deployment Target* available
>
> I need to create a VM Deployment Target
Architecture notes:
* This is creating an entity using the "VM" bundle of the Deployment Target entity type
* Fie...> As an Aegir manager,
>
> In order to make a *VM Deployment Target* available
>
> I need to create a VM Deployment Target
Architecture notes:
* This is creating an entity using the "VM" bundle of the Deployment Target entity type
* Fields:
* Target name (host)
* Routeable address (hostname or IP)
We are hoping to use ECK entities to implement this, and reproduce the behaviour of the current Platform (custom) entity type, which will eventually be replaced with Deployment Target. This means:
* Auto-create Operations and Tasks associated with the Target (see how Platform uses [AutoCreateTrait.php](profile/modules/api/src/Entity/AutocreateTrait.php))
* To use the AutoCreate code via an ECK entity type, it will probably need to be moved into a Service or similar, and attached via `preSave` or `preCreate` hooks attached to the relevant ECK entity type
* Marshal field values from the Operations and Tasks referenced by the Target (recursively), to pass them to `dispatcherd` (see how Platform uses [MarshalTrait.php](profile/modules/api/src/Entity/MarshalTrait.php)
* Similarly, this Marshaling logic likely needs to move into a Service or something to be able to be called from a hook we attach to the relevant ECK entity type.
* Base field `operation_status` (inherited on the Platform entity from [AbstractEntity.php](profile/modules/api/src/Entity/AbstractEntity.php)) should be set to the status of the most recent operation run on the entity.
* In ECK, we probably need to use [hook_entity_base_field_info()](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21entity.api.php/function/hook_entity_base_field_info/8.8.x) to attach this base field to the ECK Entity type (Deployment Target), and arrange for Operations run on the Target entity to update it, the way Platform does currently
* We aren't sure what mechanism(s) is/are best to customize ECK entity types, but the following may be useful techniques:
* https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21entity.api.php/function/hook_entity_type_alter/10
* https://www.drupaldump.com/how-swap-class-drupal-8-services
* https://www.axelerant.com/blog/drupal-8-service-decorators
* One reason to re-build the Platform functionality in parallel is to address (and ultimately remove) a lot of the boilerplate and oddities in using Entity API from D8 beta (see #76 for example)
TODO:
* [ ] Start from the existing branch for #125 which has a "cluster" ECK entity type, which we would like to rename to Deployment Target (machine name: `deployment_target`)
* **Note** we have a script in Envision which can copy/paste an existing Entity Type module tree to a new name (https://gitlab.com/consensus.enterprises/clients/envision-recycling/logistics-app/-/blob/main/web/profiles/envision/modules/copy-entity-module.sh?ref_type=heads)
* [ ] Create an Task type (bundle) for a "debug" task to attach to Deployment Targets
* [ ] Create an Operation type (bundle) for a "debug" operation to attach to Deployment Targets
* This Operation should contain a reference field to the corresponding Task type
* [ ] Arrange for Operations and Tasks to be auto-created when creating a Deployment Target entity
* [ ] Arrange for running the "debug" Operation, such that it marshals field values and dispatches them using the existing queue mechanism (similar to our [`example_update_operation`](profile/modules/examples/example_update_operation) module)Release 1 - Pluggable backends with stateless ProjectsAmbient.ImpactAmbient.Impacthttps://gitlab.com/aegir/aegir/-/issues/162Deploy a static html Project (VM deployment target)2023-11-07T15:28:29ZChristopher GervaisDeploy a static html Project (VM deployment target)> As an Aegir user,
>
> In order to deploy my static html project to a VM Deployment Target
>
> I need to run a deploy operation on my static html Project
* this is an Operation (button) whose tasks will need to run remotely on the VM...> As an Aegir user,
>
> In order to deploy my static html project to a VM Deployment Target
>
> I need to run a deploy operation on my static html Project
* this is an Operation (button) whose tasks will need to run remotely on the VM
* Marshal the variables from the Operation's tasks
* Determine the backend for each task
* Dispatch each task to the appropriate backend
* Tasks:
* Clone the git repo (from the "static html Project")
* Create a vhost in our DocumentRoot
* Render vhost template
* Copy to the server
* Restart nginx (on the VM)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/161Create a static html Project (VM deployment target)2023-11-07T15:22:53ZChristopher GervaisCreate a static html Project (VM deployment target)> As an Aegir user,
>
> In order to deploy a **static html site** to a **VM Deployment Target**
>
> I need to create a static html Project
Architecture notes:
* This is creating an entity using the "static html" bundle of the Project e...> As an Aegir user,
>
> In order to deploy a **static html site** to a **VM Deployment Target**
>
> I need to create a static html Project
Architecture notes:
* This is creating an entity using the "static html" bundle of the Project entity type
* Fields:
* hostname (must be unique)
* Git repo
* public url or embedded deploy token
* default branch (this is essentially a "singleton" project)
* Select a Deployment Target (VM)
* Output: Print instruction point the hostname towards the site (deployment target IP address or hostname)Release 1 - Pluggable backends with stateless Projectshttps://gitlab.com/aegir/aegir/-/issues/160As an *Aegir5 site-hoster*, I want to to "migrate" my site to the latest vers...2023-10-18T16:37:34ZChristopher GervaisAs an *Aegir5 site-hoster*, I want to to "migrate" my site to the latest version of an Application* As an *Aegir5 site-hoster*, I want to to "migrate" my site to the latest version of an Application* As an *Aegir5 site-hoster*, I want to to "migrate" my site to the latest version of an ApplicationRelease 4 - Application workflows