Read zero-padded version segments as decimal numbers

José M. Requena Plensrequested to merge
jmrp/semver_dialects:jmrp-semver-dialects-decimal-segments into master

What does this MR do?

SemanticVersion, which the pypi, packagist and conan dialects use, reads each numeric segment with Integer() and no base. Without a base, Ruby treats a leading 0 as an octal prefix, so 010 reads as 8, and 08 and 09 are not numbers at all, which ends the numeric part of the version at that segment. Versions with a segment such as 08, 09 or 010 therefore compare wrongly (00 to 07 happen to come out right):

  • 2023.08.01 sorts before 2023.07.01
  • 1.010 equals 1.8
  • 2023.1.6 is outside >=2021.04.11,<2023.09.24
  • 0.01 satisfies >=0.08

This MR passes base 10 to both Integer() calls, in String#number? (lib/utils.rb) and in SemanticVersionSegment#initialize. That is how the three ecosystems read these segments: PEP 440 normalizes every integer through int(), and Python's packaging, composer/semver and Conan's own Version all return what this branch returns for every pair I tried (table below).

Two side effects are visible, and I think both are right:

  • String#number? no longer accepts a 0x, 0b or 0o prefix. Inside a version these never reach it, because the parser splits letters from digits first, and the GitLab application does not call String#number? itself.
  • SemanticVersion#minor and #patch of 2023.08.01 return "08" and "01" instead of "0". #major is unchanged.

The yt-dlp document >=2021.04.11,<2023.09.24 in expectation/version_sat/pypi.yaml now holds, so I marked it resolved. I evaluated all 79 expectation documents on master and on this branch, and that is the only one whose verdict changes. The four other confirmed pypi documents that already hold on master are left as they are.

One thing I noticed while doing that: spec/semver_dialects_spec.rb builds the expectation path with File.expand_path('../..', __dir__), which points at the directory above the repository since the spec moved out of spec/unit/ in 251b0d10, so the suite loads no expectation document today. That is a separate defect and I left it out of this MR; the evidence below includes a run with the path corrected.

The version goes to 4.1.4 in version.rb, CHANGELOG.md and Gemfile.lock, as in !128 (merged).

For context on reach: GitLab pins semver_dialects 4.1.3 (~> 4.1), and continuous vulnerability scanning matches advisory ranges with IntervalSetParser.parse and parse_version (affected_version_range_matcher.rb). With 4.1.3, yt-dlp 2023.1.2, 2023.1.6, 2023.03.04 and 2023.07.06 do not match the range of CVE-2023-40581 (>=2021.04.11,<2023.09.24); with this branch they do, as packaging says they should.

What are the relevant issue numbers?

None. I found this by comparing the gem's ordering with each ecosystem's own library, and I did not find an open issue for it.

Does this MR meet the acceptance criteria?

Comparisons: master, this branch, and each ecosystem's library

a <=> b through SemverDialects.parse_version. The master and this branch columns are the same for pypi, packagist and conan. The libraries are packaging 26.3, composer/semver 3.5.1 (after VersionParser::normalize) and Conan 2.33.0.

a b master this branch packaging composer/semver Conan
2023.08.01 2023.07.01 -1 1 1 1 1
2023.09.24 2023.9.24 -1 0 0 0 0
1.010 1.10 -1 0 0 0 0
1.010 1.8 0 1 1 1 1
1.010 1.9 -1 1 1 1 1
2024.08 2024.7 -1 1 1 1 1
2018.02.08 2018.2.8 -1 0 0 0 0
0.08 0.1 -1 1 1 1 1
1.07 1.7 0 0 0 0 0

version_satisfies?('pypi', version, range) against SpecifierSet(range).contains(version):

version range master this branch packaging
2023.08.01 <2023.07.15 true false False
1.010 <1.9 true false False
2023.1.6 >=2021.04.11,<2023.09.24 false true True
2023.10.01 >=2021.04.11,<2023.09.24 false false False
0.01 >=0.08 true false False

composer/semver also answers Semver::satisfies('2023.08.01', '<2023.07.15') with false, and both composer/semver and Conan's VersionRange answer false for 0.01 against >=0.08.

Specs, lint and version verification

Run in the ruby:3.2 (3.2.11), ruby:3.3 (3.3.12) and ruby:3.4 (3.4.11) images, amd64, with the lockfile frozen:

  • bundle exec rspec: 1492 examples, 0 failures on each (1464 on master).
  • bundle exec rubocop: 45 files inspected, no offenses detected on each.
  • ruby scripts/version_verification.rb and ruby scripts/version_verification.rb v4.1.4: pass on each.

The new specs with master's lib/utils.rb and lib/semver_dialects/semantic_version.rb: 1492 examples, 26 failures on each version. The two new examples that pass there are '010'.number? (true either way) and #major of 2023.08.01.

Expectation documents

Evaluating every document in expectation/version_sat/*.yaml directly with version_satisfies? (79 documents, on master 64 resolved, 12 confirmed and 3 dismissed): all 64 resolved documents hold on master and on this branch, and the yt-dlp >=2021.04.11,<2023.09.24 document goes from failing (2023.1.6, 2023.01.06, 2023.1.2 and 2023.01.02 reported outside the range) to holding.

With root_dir = File.expand_path('..', __dir__) in a local copy, the suite runs 310 more examples: on this branch 1802 examples, 0 failures, 14 pending; with master's two library files, the four includes examples of the yt-dlp document fail along with the new specs. master itself with only the path corrected gives 1769 examples, 0 failures, 15 pending.

Merge request reports