Enabling modem/IPA causes shutdown to hang for a *very* long time

Sometimes it takes up to 72 seconds to finally kill the modem's remoteproc modules so the system finally reboots. This is the case even with e.g. reboot -f and can cause the command to SEGFAULT.

Example of dmesg -w during shutdown with ipa probed:

...
[  491.715262] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  491.771350] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  491.828604] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  491.884364] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  491.940347] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  491.945622] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  491.996350] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.001664] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  492.052347] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.057609] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  492.108347] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.113618] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  492.164617] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.169988] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  492.220349] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.225612] ipa 1e40000.ipa: channel 7 global error ee 0x00000000 code 0x00000002
[  492.276457] ipa 1e40000.ipa: GSI command 2 for channel 7 timed out, state 4
[  492.281756] ipa 1e40000.ipa: error -11 attempting to stop endpoint 15
[  492.282166] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.332575] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.337859] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.388431] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.393287] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.444353] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.449520] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.500843] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.506216] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.556274] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.561541] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.612433] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.617730] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.668357] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.673623] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.724260] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.729385] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.780263] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.785499] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.836267] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.841272] ipa 1e40000.ipa: channel 6 global error ee 0x00000000 code 0x00000002
[  492.892369] ipa 1e40000.ipa: GSI command 2 for channel 6 timed out, state 4
[  492.897527] ipa 1e40000.ipa: error -11 attempting to stop endpoint 22
[  492.899242] ipa 1e40000.ipa: channel 6 bad state 4 before reset
[  492.905516] ipa 1e40000.ipa: channel 7 bad state 4 before reset
[  492.907251] ipa 1e40000.ipa: channel 7 bad state 4 before reset
[  492.937284] ipa 1e40000.ipa: channel 7 bad state 4 before dealloc
[  492.954293] ipa 1e40000.ipa: channel 6 bad state 4 before dealloc