Skip to content

Two more functions to chunk code using the prompt engine.

Hongtao Yang requested to merge chunking-functions into main

This MR implements chunking functions other than the default Poisson chunker. Specifically, this MR implements:

  1. CurrentFunctionChunker, this chunker corresponds to method 2 of https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180#methods-of-chunking
  2. CurrentContextChunker, this chunker corresponds to method 3 of https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180#methods-of-chunking

NOTE: the chunking functions and parsing functions in this MR only works with python code. We'll add support for other languages in another MR.

ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180, https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/4

Edited by Hongtao Yang

Merge request reports