@larswirzenius guided me through getting check to pass on a debian VM and during the process I noticed that the lmodern package doesn't seem to be installed after doing dpkg-builddeps. I had to install it manually.