Unit Test Server and Client Set up
Overview
Follow up:
Hi @squadri , yes you are right. Implementing the unit test server is important but I think that needs to be in a dedicated issue to track.
Its practical to have the first version of the dataset in this iteration, and start building the server in the next iteration. WDYT?
@HongtaoYang Thanks @squadri , I think we can drop Add more configuration options to the MCP client and just add any missing options when the needs arise.
Right now after we address the bug around GitLab-Code-Parser project having incorrect filename, we should move towards the client setup to run this dataset.
In my opinion we can close #549733 (closed) and #508167 (closed), then open new issues to track the client setup. WDYT?
Background
This issue follows up on the Code Generation Dataset POC that created a more realistic testing environment for code generation by simulating real-world enterprise codebases rather than isolated code snippets.
In the previous issue, we implemented an MCP server and client framework to auto-generate realistic evaluation datasets from various GitLab projects (i.e. GitLab-AIGW, GitLab-Code-Parser, etc.) and uploaded the first example datasets to Langsmith.
Objectives
- Implement a client to invoke code suggestions API with the evaluation datasets to generate responses for tasks that will be verified by the Unit Test Server.
- Create a functional testing framework (i.e. Unit Test Server) for LLM generated code during evaluations
Success Criteria
- API invocation client is implemented
- Unit tests can validate the functionality of generated code
- Process is documented
Success Criteria
- API invocation client is implemented
- Unit tests can validate the functionality of generated code
- Process is documented