Remove limitations of variable substitution in Prometheus proxy API

Problem to solve

Follow-up issue from !19994 (comment 244881805).

The Ruby % operator is used to perform variable substitution on PromQL queries. For example, with up{env="%{ci_environment_slug}"} as query, the ci_environment_slug variable will get replaced with the environment's slug before the query is sent to Prometheus for processing.

However, Ruby's % operator has a couple of problems:

1. Ruby's % operator does not replace variables if there are variables in the string with no match in the hash.

   For example: '%{a} %{b}' % { a: 'a' } will return '%{a} %{b}' instead of a %{b}.

2. Additionally, Ruby's % operator will throw an exception with strings like %{a} or %a.

Intended users

Further details

Proposal

We can try to find alternate templating engines to use. However, a different templating engine might need all dashboards to be modified if the engine does not support the current variable interpolation format (%{}).

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references