Draft: feat: add chaos endpoint for FF observability

Relates to issue #594205

Changes

This MR implements a chaos test endpoint with feature flag integration for observability testing.

✅ Task 1: Feature Flag Definition

• File: config/feature_flags/development/ebonet_chaos_tests.yml
• Created a development-type feature flag named ebonet_chaos_tests with default_enabled: false

✅ Task 2: Chaos Controller Method

• File: app/controllers/chaos_controller.rb
• Added chaos_test method that:
  • Selects a random project from the first 1000 projects
  • Checks if the ebonet_chaos_tests feature flag is enabled for that project
  • Randomly introduces chaos (20% chance of 500 error, 20% chance of 300ms delay)
  • Returns OK otherwise

✅ Task 3: Route Configuration

• File: config/routes.rb
• Added get :chaos_test route within the chaos resource block
• Endpoint available at /-/chaos/chaos_test

✅ Task 4: RSpec Tests

• File: spec/controllers/chaos_controller_chaos_test_spec.rb
• Comprehensive test coverage including:
  • Feature flag disabled scenario (returns OK)
  • Feature flag enabled with random failure (returns 500)
  • Feature flag enabled with random delay (sleeps 300ms)
  • Feature flag enabled with no chaos (returns OK)
  • No project found scenario (returns OK)

✅ Task 5: Documentation

• File: doc/development/chaos_endpoints_observability.md
• Complete documentation covering:
  • Overview and purpose of the chaos_test endpoint
  • How to enable and use the endpoint
  • Feature flag management
  • Multiple approaches for generating test requests (bash loops, scripts, Apache Bench)
  • Guidance on creating Kibana dashboards with specific visualizations
  • Prometheus query examples
  • Best practices and example testing workflow

Testing

The implementation includes comprehensive RSpec tests that verify all scenarios including feature flag behavior, random chaos injection, and edge cases.

Documentation

Full documentation is provided in doc/development/chaos_endpoints_observability.md with examples for Kibana dashboards, Prometheus queries, and testing workflows.