Skip to content

Investigate into logging architecture for workspace

MR: Pending

Description

Update 13th Aug

After the investigation issue for workspace events, we have identified below 5 aspects below.

  • Sending data reliably from Agentk to Rails.
  • Rails processing this data and generating GitLab Events data.
  • GitLab Events data getting persisted in the DB.
  • Backend APIs for fetching the persisted data from the DB.
  • Frontend integration

The current plan is to analyze design and implementation for top 3 points first. Then look into the Rails API and Frontend.

Initial description

After some investigation, we have had thoughts on how industry implement logging from workspace to its UI. In this issue we will explore and decide what should be the architecture of workspace loggings.

For logs, we generally category them into three types,

  1. workspace building/creation logs
  2. workspace agent logs when workspace is created successfully
  3. audit logs is for user actions, and are generally saved into DB (we have some issues for this, but this is quite unrelated to our current epic)

Our major focus for the current epic should be 1 (or maybe 2 as well?).

Some rough thoughts so far,

  • Logs could be structured entities from our scripts (like, fetching projects from ..., SSH is ready), and also unstructured if we would like to expose tools logs as well (like git clone logs from git command).
  • Those logs should be stored into persistent storage in k8s, and not be persistent in GitLab backend DB.
  • We could use GitLab backend as a bridge (for example when we have workspace from GitLab runner and k8s), and directly expose API from workspace, for getting log contents.
  • Web Socket and long-lived HTTPS is both used in some cases.

Will dig deeper and try bring up some solutions for discussions.

References

Acceptance criteria

TODO: Fill out (required)

  • [Describe what must be achieved to complete this issue.]
  • [If applicable, please provide design specifications for this feature/enhancement.]
  • [If applicable, please list any technical requirements (performance, security, database, etc.)]

Implementation plan

TODO: Fill out or delete (optional)

[Provide a high-level plan for implementation of this issue, including relevant technical and/or design details.]

Edited by zli