[Research] Consider potential storage options for custom dashboards

Problem to solve

The existing customizable analytics dashboards implementation uses the project repo to commit changes to dashboards. We then read the project repo's special directories structure to find dashboards and visualizations to render.

This structure is unlikely to work when we move to having dashboards outside the group → project hierarchy.

We need to determine:

• Potential solutions to this problem
• Pro's and con's of each solution
• Potential costs (time/maintenance/reliability/customizability) for each solution
• Factor in migrating analytics dashboards to the implementation

Two existing ideas:

1. Store dashboards in a new PostgreSQL table
2. Store in object storage using Minio.

Requirements

Functional Requirements

• Platform Compatibility: Must work on both SaaS and customer instances (self-hosted, dedicated).
• Access Control:
  • Prevent unauthorized access to dashboards.
  • Support dashboard sharing and fine-grained permission management across different organizational levels (e.g., group, project, user, external sharing).
• Data Hierarchy: Support dashboard configurations that exist outside the traditional group > project hierarchy.
• Auditing and Versioning:
  • Provide an auditable and traceable log of all changes (persistent change events).
  • Support the ability to roll back a dashboard to a previous version.
• Concurrency: Safely handle concurrent write operations from multiple users to prevent conflicts and data corruption.
• User Experience: Provide useful and clear error messages to the end-user.

Non-Functional Requirements

• Conflicts: Optimistic locking to prevent users from saving a change over the change of another user similar to when editing issues.
• Performance: API responses for dashboard configurations must be within a 200-500ms range to ensure a responsive user experience.
• Scalability: The solution must be able to scale to handle multiple dashboard configurations, with consideration for the maximum number of dashboards; as well as the maximum size of each configuration file.
• Availability and Recovery:
  • Must have a clear and automated plan for backup and disaster recovery (DR).
  • The system should be designed for high availability (HA), with zero-downtime updates and locally saving state in case of outages.
• Security and Compliance:
  • Dashboard configurations should be treated as sensitive data, potentially containing private information.
  • The solution should support data governance needs, such as data residency and the ability to produce audit trails for compliance with regulations like GDPR or HIPAA.
• Monitoring: The system must be monitorable to provide insights into cost, usage, and performance.

Future and Architectural Requirements

• Advanced Search: The solution must eventually support complex searching and filtering operations on dashboard configurations (e.g., searching for dashboards containing specific panel types or metrics).
• Zero-Downtime Schema Evolution: Must support the ability to make changes to the dashboard configuration schema itself (e.g., adding or renaming fields within the JSON/YAML structure) without causing application downtime.
• Developer Experience: The chosen solution should integrate well with the existing technology stack (Ruby on Rails, Vue, GraphQL) and have a strong ecosystem of tools and community support to minimize technical debt and improve long-term maintainability.
• Real-Time Capabilities (Optional but considered): The architecture should ideally not prevent the future implementation of real-time features, such as live collaboration on dashboards.

Definition of done

1. Review potential solutions and pick the path forward
2. Create an issue to document the architectural choice as part of https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/dashboard_customization_framework/
3. Create an issue to implement the chosen solution as part of the customization component.