Fix issues in record_size_limit extension handling

This fixes the issues discovered in #676 (closed), except the HRR tests (see #617).

Checklist

• Code modified for feature
• Test suite updated with functionality tests
• Test suite updated with negative tests
• Documentation updated / NEWS entry present (for non-trivial changes)

Reviewer's checklist:

• Any issues marked for closing are addressed
• There is a test suite reasonably covering new functionality or modifications
• Function naming, parameters, return values, types, etc., are consistent and according to CONTRIBUTION.md
• This feature/change has adequate documentation added
• No obvious mistakes in the code