Resolve "QDGGA: Integrate into distributed pipeline"
Note: this ticket uses the entire cluster which is a global resource. When testing it is necessary to ping in the #general chat and make sure noone is running a long running job that could be affected by spinning up GPU resources.
pre
- ask the discord server if you are clear to spin up the cluster
- ssh into devops@niagara.cs.rit.edu. if you need the creds ask matt
part 1
- cd to
/home/devops/services/MathSeer-extraction-pipeline
- run
make clean
- run
make clean-repos
- run
make
- run
make -f DM.mk swarm-up
- run
make -f DM.mk swarm-run-example
- take note of the time it takes to complete, and report in MR thread
- run
make -f DM.mk swarm-down
part 2
-
run
make example
-
take note of how long it takes, and report in MR thread
-
run
./bin/compare-outputs-sync-async
and confirm differences are minimal. if you are not sure if the differences are acceptable you can post them in the MR thread.Note that qdgga is not compared. This is because the values are not lining up. This will be addressed in the server ticket on the qdgga repo -
run this command to get the output files onto a computer in which you can view them in a browser. The SLT representation will be absent, that is fine. it will save the directory in ur current working path
scp -r devops@niagara.cs.rit.edu:/home/devops/services/MathSeer-extraction-pipeline/outputs ./outputs