Skip to content

Pdf size sorting

Matt Langsenkamp requested to merge pdf_size_sorting into master

This MR isnt accompanied by an actual ticket as it was just a branch i made to run a quick experiment which turned into a handful of smaller refactors around how the pipeline runs makes requests to the services.

Note: this ticket uses the entire cluster which is a global resource. When testing it is necessary to ping in the #general chat and make sure noone is running a long running job that could be affected by spinning up GPU resources.

pre

  • ask the discord server if you are clear to spin up the cluster
  • ssh into devops@niagara.cs.rit.edu. if you need the creds ask matt

part 1

  • cd to /home/devops/services/MathSeer-extraction-pipeline
  • run make clean
  • run make clean-repos
  • run make
  • run make -f DM.mk swarm-up
  • run make -f DM.mk swarm-run-example
  • take note of the time it takes to complete, and report in MR thread
  • run make -f DM.mk swarm-down

part 2

  • run make example

take note of how long it takes, and report in MR thread

run ./bin/compare-outputs-sync-async and confirm differences are minimal. if you are not sure if the differences are acceptable you can post them in the MR thread.Note that qdgga is not compared. This is because the values are not lining up. This will be addressed in the server ticket on the qdgga repo

run this command to get the output files onto a computer in which you can view them in a browser. The SLT representation will be absent, that is fine. it will save the directory in ur current working path scp -r devops@niagara.cs.rit.edu:/home/devops/services/MathSeer-extraction-pipeline/outputs ./outputs

Edited by Matt Langsenkamp

Merge request reports