Convert AST to postgres query

Convert the search AST to a postgres query

The required methods according to the concern to implement is process and transform and then we need to implement processing for the node types:

  • filter
  • prefix
  • or
  • and
  • knn
  • limit

These will convert to postgres queries containing WHERE clauses in different ways.

https://gitlab.com/gitlab-org/gitlab/-/blob/master/gems/gitlab-active-context/lib/active_context/query/processor_example.rb might be good for an example, maybe a lot of this can be reused.

We use cosine similarity (<=>): https://github.com/pgvector/pgvector?tab=readme-ov-file#querying

Suggested development:

  1. Create a postgres db, e.g. docker run -p 5432:5432 --name pgvector17 -e POSTGRES_PASSWORD=password pgvector/pgvector:pg17
  2. Create a connection: Ai::ActiveContext::Connection.create!(active: true, name: 'pg1', options: {port: 5432, host: 'localhost', username: 'postgres', password: 'password'}, adapter_class: 'ActiveContext::Databases::Postgresql::Adapter', prefix: 'active_context')
    1. Make this the active connection
  3. Create a migration to create a table
class CreateMergeRequests < ActiveContext::Migration[1.0]
  milestone '17.9'

  def migrate!
    create_collection :merge_requests, number_of_partitions: 3 do |c|
      c.bigint :issue_id, index: true
      c.bigint :namespace_id, index: true
      c.prefix :traversal_ids
      c.vector :embeddings, dimensions: 768
    end
  end
end
  1. Run the migration: ActiveContext::Migration::Dictionary.migrations.last.new.migrate!
  2. Add the extension by connecting to psql: CREATE EXTENSION vector;
  3. Now you can use this table for writing and testing sql queries
Edited by Arturo Herrero