Two more functions to chunk code using the prompt engine.
This MR implements chunking functions other than the default Poisson chunker. Specifically, this MR implements:
-
CurrentFunctionChunker
, this chunker corresponds to method 2 of https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180#methods-of-chunking -
CurrentContextChunker
, this chunker corresponds to method 3 of https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180#methods-of-chunking
NOTE: the chunking functions and parsing functions in this MR only works with python code. We'll add support for other languages in another MR.
ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/180, https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/4
Edited by Hongtao Yang