Don't build with -Wcast-align

This is a useless warning on any architecture with a barrel shifter,
which includes Intel and ARM and basically everything nowadays. Even
so, we'd enable it out of perfectionism, but the GCC directives that
ought to be useful for forcing structure alignment in order to
suppress it locally don't seem to be working quite right.