Pdf size sorting
This MR isnt accompanied by an actual ticket as it was just a branch i made to run a quick experiment which turned into a handful of smaller refactors around how the pipeline runs makes requests to the services.
Note: this ticket uses the entire cluster which is a global resource. When testing it is necessary to ping in the #general chat and make sure noone is running a long running job that could be affected by spinning up GPU resources.
pre
- ask the discord server if you are clear to spin up the cluster
- ssh into devops@niagara.cs.rit.edu. if you need the creds ask matt
part 1
- cd to /home/devops/services/MathSeer-extraction-pipeline
- run
make clean
- run
make clean-repos
- run
make
- run
make -f DM.mk swarm-up
- run
make -f DM.mk swarm-run-example
- take note of the time it takes to complete, and report in MR thread
- run
make -f DM.mk swarm-down
part 2
- run
make example
take note of how long it takes, and report in MR thread
run ./bin/compare-outputs-sync-async and confirm differences are minimal. if you are not sure if the differences are acceptable you can post them in the MR thread.Note that qdgga is not compared. This is because the values are not lining up. This will be addressed in the server ticket on the qdgga repo
run this command to get the output files onto a computer in which you can view them in a browser. The SLT representation will be absent, that is fine. it will save the directory in ur current working path scp -r devops@niagara.cs.rit.edu:/home/devops/services/MathSeer-extraction-pipeline/outputs ./outputs