Cleanup gem files

Original deb size: 661M

deb size after merge: 631M (package in CI)

I had hoped to reduce the size more, but most of the trimming mentioned in #3236 (closed) has already been done with earlier merges

Closes #3236 (closed) and #3414 (closed).