Properly handle the restricted ref frames

This CL allows the switch frame to be an inter frame. It further allows the subsequent inter frames to use reference frames before the S frame as restricted reference frames.

This resolves issue #1332 (closed) and #1333 (closed).