Skip to content

Implement LangSmith to Repository Sync for Datasets

Description

Currently, we need to implement a system to sync datasets from LangSmith to our main repository. This is necessary to ensure that changes made in LangSmith, where people are actively working, are reflected in our main data storage.

Objectives

  1. Develop a mechanism to detect changes in LangSmith datasets.
  2. Create a process to sync these changes to the main repository.
  3. Ensure the sync process can handle both new datasets and modifications to existing ones.
  4. Consider versioning for datasets, possibly using tagged commits in the repository.

Considerations

  • LangSmith doesn't currently have webhooks for dataset changes, so we may need to implement a polling system.
  • We need to handle both CSV and JSONL formats, as LangSmith UI accepts CSV while our upload script uses JSONL.

Questions to address

  1. How frequently should we check for updates in LangSmith?
  2. How do we handle conflicts if changes are made in both LangSmith and the repository?
  3. Do we need to implement any approval process before syncing changes to the main repository?

Next steps

  • Investigate LangSmith API capabilities for detecting dataset changes.
  • Schedule a sync meeting with relevant team members to discuss the best approach.

Related issues

Edited by David O'Regan