Skip to content

Auto-evaluate: Pipelining

General idea: Run segmentation/evaluation in parallel when auto-evaluate is checked

Workflow

  • After a motion recording in the dataset was segmented, the SegmentationController will report the individual SegmentationResult in a topic
  • The UI subscribed to the topic, and if auto-evaluate was checked, it will immediately begin to evalute the SegmentationResult one-by-one in parallel (while still segmenting)
  • The UI caches the individual EvaluationResults
  • With the last SegmentationResult being evaluated, the UI will trigger the total evaluation by providing the vector of cached EvaluationResults to the EvaluationController (basically we just need to make the corresponding function in the EvaluationController public, but see (pseudo-)code below)
  • The EvaluationController will then report the EvaluationResult as usual for the UI to display.
  • We could make a new function in the EvaluationController for this so the UI doesn't have to assembly the expected EvaluationResult. E.g.:
armarx::EvaluationResultPtr
armarx::EvaluationController::evaluateTotalResultAndPack(std::vector<armarx::EvaluationResult> evaluationResults)
{
    armarx::EvaluationResultPtr result = new armarx::EvaluationResult();
    result->name = "Evaluation results";
    // isGroup, isGroupExpanded, blah blah

    // Push total result to "Evaluation results"
    result->subResults.push_back(armarx::EvaluationController::evaluateTotalResult(evaluationResults);

    // Initialize "Individual results" group and push evaluationResults into its subResults    
    armarx::EvaluationResultPtr individiaulResults = //...
    individualResults->subResults.push_back_vector(evaluationResults); // I wish C++ could do that. Use .insert, end(), begin(), end()...

    // Push "Individual results" to "Evaluation results"
    result->subResults.push_back(individualResults);

    return result;
}

Possible caveats

  • reportEvaluationProgress must be tweaked in order to push the right progress to the evaluation progress bar in the UI
    • Possible fix: Each SegmentationResult struct could have two new members: int resultNumber and int resultsTotal, where 1 <= resultNumber <= resultsTotal and resultsTotal = segmentationResults.size(). The EvaluationController can make use of that then.
  • The auto-evaluate checkbox needs to be disabled when the segmentation was started, otherwise it could miss results
    • Possible fix: Cache SegmentationResults anyway => No need to disable checkbox. But then, what if the user changes his mind again? Dismiss EvaluationResults?
    • Possible fix: Remove checkbox and replace it with a new button Segment & Evaluate