I am writing a compiler that uses Rug for the language's arbitrary precision integer type. To ensure that the grammar of the language would be compatible with Rug and GMP's parsing algorithms, I've inspected the source of both, since the docs are ambiguous. In the process, I've found some inconsistencies between the Rug and GMP parsers and I have some suggestions for extending Rug's parsing API for use cases with pre-validated input. As I am only using `Integer`, I mostly looked there, but this should be generalizable to the other types. Here's what I found: ## GMP GMP has two functions for converting from “strings”: the lower-level, base-n-digit-oriented [`mpn_set_str`](https://gmplib.org/manual/Low_002dlevel-Functions#index-mpn_005fset_005fstr) in `mpn/generic/set_str.c`, which Rug uses, and the character-string-oriented [`mpz_set_str`](https://gmplib.org/manual/Assigning-Integers) in `mpz/set_str.c`. ### `mpn_set_str` `mpn_set_str` takes as input a non-zero-length `const unsigned char *` byte array and interprets each byte as a digit in the given base in the range 0..base (i.e., not an ASCII character); no range checking is performed. It has a fast path for bases that are powers of 2 and uses constants for the common base of base 10, to allow for better compiler optimizations. ### `mpz_set_str` `mpz_set_str` takes as input a null-terminated `const char *` string, parses it using a temporary buffer, then delegates to `mpn_set_str`. It supports values 0..=62 for `base`. Bases 37..=62 use the sequence `0-9A-Za-z` for its alphabet and bases 1..=36 use the same, but are case-insensitive. When `base == 0`, it detects the base from a C-style prefix for 2, 8, or 16 or else 10. Although the docs state otherwise, `base == 1` is actually not forbidden, but it will always yield a zero value without allocating, since all digits are stripped as leading zeros. The grammar for `mpz_set_str` is as follows: - `base != 0`: `\s*-?[0-9a-zA-Z][0\s]*[0-9a-fA-F\s]*` (digits are checked to be in range) - `base == 0`: - with a binary prefix: `\s*-?0[bB][0\s]*[01\s]*` - with an octal prefix: `\s*-?0[0\s]*[0-7\s]*` - with a hex prefix: `\s*-?0[xX][0\s]*[0-9a-fA-F\s]*` - without a prefix: `\s*-?[1-9][0\s]*[0-9\s]*` where I've used `\s` to denote `[ \t\n\v\f\r]`. A quirk that I noticed is that whitespace is allowed everywhere, aside from between `-` and the first digit, which may be a bug; (whitespace is also not allowed within the `0[xXbB]` prefixes, but that makes sense). It returns `0` if the input is well-formed or returns `-1` if `base > 62` or a digit is out of range for `base`. ## Rug `Integer::parse_radix` `Integer::parse_radix` fills the same role as `mpz_set_str`—converting the input ASCII to the format for `mpn_set_str`—, but rolls its own parsing algorithm and has subtle differences: `radix` must be in 2..=36, so `radix == 0` prefix detection, `radix == 1`, and `radix` in 37..=62 are not supported. The grammar: `\s*[+-]?\s*[0-9a-zA-Z][0-9a-zA-Z_\s]*` (digits are checked to be in range and `\s` identically represents `[ \t\n\v\f\r]`) Aside from omitting radix detection with a `0[xXbB]` prefixes, the grammar is slightly more permissive than `mpz_set_str`: A positive `+` sign is optionally allowed, whitespace is allowed between the sign and the first digit, and underscores are allowed anywhere after the first digit. Error reporting is barely more expressive: it distinguishes between `InvalidDigit` and `NoDigits` errors with the bare enum `ParseErrorKind`. Invalid `radix` is a panic. ## Suggestions for Rug ### API I would like to see an API directly exposed for `mpn_set_str` in Rug. In my parser, I am already validating the input according to my language's grammar. Since it differs from that of `Integer::parse_radix`, I need to allocate a temporary buffer to normalize it for `Integer::parse_radix`, which then allocates another temporary buffer to convert it for `mpn_set_str`. I've annotated changes to `integer` inline: ```rust impl Integer { #[inline] pub fn parse_radix<S: AsRef<[u8]>>( src: S, radix: i32, ) -> Result<ParseIncomplete, ParseIntegerError> { parse(src.as_ref(), radix) } // Since the radix is so tied to the representation of `digits`, I don't // think an Integer::parse_raw is necessary. // NEW #[inline] pub unsafe fn parse_radix_raw<S: AsRef<[u8]>>( digits: &'a S, radix: i32, is_negative: bool, ) -> ParseRawIncomplete<'a> { ParseRawIncomplete { is_negative, digits: digits.as_ref(), radix, } } // NEW // To generalize Assign over the differing ownership: fn assign_from_parse_incomplete<'a>( &mut self, digits: &'a [u8], radix: i32, is_negative: bool, ) { if digits.is_empty() { xmpz::set_0(self); return; } xmpz::realloc_for_mpn_set_str(self, digits.len(), radix); unsafe { let size = gmp::mpn_set_str(self.inner.d.as_ptr(), digits.as_ptr(), digits.len(), radix); self.inner.size = (if is_negative { -size } else { size }).unwrapped_cast(); } } } // For pre-validated inputs, this should all ideally be skipped: fn parse(bytes: &[u8], radix: i32) -> Result<ParseIncomplete, ParseIntegerError> { // (Simplified for terseness) assert!((2..=36).contains(&radix), "radix out of range"); let mut digits = Vec::with_capacity(bytes.len()); for &b in bytes { // ... } Ok(ParseIncomplete { is_negative, digits, radix, }) } #[derive(Debug)] pub struct ParseIncomplete { is_negative: bool, // This is owned, but it doesn't need to be digits: Vec<u8>, radix: i32, } // NEW // For API compatibility, this needs to be a separate struct: #[derive(Debug)] pub struct ParseRawIncomplete<'a> { is_negative: bool, // Avoids a clone! digits: &'a [u8], radix: i32, } impl Assign<ParseIncomplete> for Integer { fn assign(&mut self, src: ParseIncomplete) { self.assign_from_parse_incomplete(src.digits.as_slice(), src.radix, src.is_negative) } } // NEW impl Assign<ParseRawIncomplete> for Integer { fn assign(&mut self, src: ParseRawIncomplete) { self.assign_from_parse_incomplete(src.digits, src.radix, src.is_negative) } } ``` `Rational` should be simple and mirror the changes for `Integer`: ```rust #[derive(Debug)] pub struct ParseIncomplete { is_negative: bool, digits: Vec<u8>, den_start: usize, radix: i32, } ``` For `Float`, it's a matter of adding an enum variant for a borrowed `CStr`. Unfortunately, `mpfr_strtofr` depends on the NUL-terminated property, so it's better to allow the caller to avoid a clone for that. API already exists to constructed a float from a `Special` and `NegNan` just negates that. ```rust #[derive(Debug)] pub enum ParseIncomplete<'a> { CString { c_string: CString, radix: i32 }, // Add a borrowed variant: CStr { c_str: &'a CStr, radix: i32 }, Special(Special), NegNan, } ``` `Complex` would be done in terms of `Float`: ```rust #[derive(Debug)] pub enum ParseIncomplete { Real(FloatParseIncomplete), Complex(FloatParseIncomplete, FloatParseIncomplete), } ``` ----- As for `mpz_set_str`, I don't think that it needs to be exposed in Rug. Since the grammar is incompatible, it couldn't be a drop-in replacement for the logic in `Integer::parse_radix` and the differences are subtle enough that it's probably not worth having as a complementary method. ### Other Some small optimizations could be borrowed from `mpz_set_str` for `Integer::parse_radix`: - Using a table for looking up digits for bytes instead of matching, to avoid branching (see the table `__gmp_digit_value_tab` in `mp_dv_tab.c`). - Allocating the intermediate buffer after trimming whitespace and leading zeros to reduce the allocated length and avoid an allocation entirely if the input matches `[0\s]*` (`mpz_set_str` only trims leading whitespace, but that's a limitation of C `strlen`; we can do trailing too). ## Alternatives Using `gmp-mpfr-sys` and `Integer::from_raw`. The API is *almost* already in Rug though, so it would be nice to add the extra bits and it would make it more discoverable and documented for those with similar use cases. ## Contributing I'd be happy to submit a merge request with these features, if this is something you'd want added. It's hard to get the code right when writing it out “on paper”, like I've done above, so there's likely a better way. In fact, it's possibly better to do `integer::ParseIncomplete` like how I drafted `float::ParseIncomplete`, by turning it into an enum and having variants for owned and borrowed.