Improve broken regex library tests.

Xcode 9.3's regex library seems to be completely broken, and is causing crashes (by their default-enabled stack-smashing detection).

Christian Ebert reported the library also doesn't seem to recognize subexpressions during regcomp, so add a detailed test which will hopefully filter the implementation out.