Fix XML character references not being decoded in cell values and attributes (regression in 9.3.0)
First I wanted to thank you for your work !
What's broken
Since 9.3.0 (migration from the DOM parser to the SAX parser), XML character references and predefined entities come through literally in parsed data:
// A cell containing "Café", in a file written by openpyxl
// (openpyxl escapes all non-ASCII text as numeric character references):
readXlsxFile(...) // → "Café" instead of "Café"Affected: any .xlsx whose XML escapes characters as references. openpyxl-generated files are the most visible case (all non-ASCII text is escaped), but files written by Excel itself are affected too wherever &, < or " appear in values — including sheet names (<sheet name="P&L"/> is returned as P&L).
Reading a real-world openpyxl-generated file here (54 data rows): 101 cells contain literal &#...; references on master, 0 with this fix.
Root cause (two layers)
-
source/xml/parseXmlStream.saxen.js—saxendoesn't decode character references itself; by design it passes adecodeEntities()function to itstext/openTaghandlers for the consumer to call. Theontexthandler ignored that argument, and attribute values were passed through raw as well. -
source/saxen/parser.js— the copy-pastedsaxenwrapper hadreturn Parser(options)at the top of the function body. Function declarations are hoisted, so the parser worked, but none of the top-levelvarinitializers below thereturnever ran:ENTITY_PATTERN,ENTITY_MAPPING,fromCharCode, … all stayedundefined.decodeEntities()was therefore a silent no-op even when called —s.replace(undefined, …)searches for the literal string"undefined"and matches nothing. (The other, currently unused copyparseXmlStream.saxen.code.jsalready has thereturnat the bottom; this MR mirrors it.)
Changes
source/saxen/parser.js— movedreturn Parser(options)to the bottom of the wrapper so thevarinitializers actually run. This also fixesNAME_CACHE,NON_WHITESPACE_OUTSIDE_ROOT_NODEand thensUri = decodeEntities(value)path, which were all silentlyundefinedfor the same reason.source/xml/parseXmlStream.saxen.js— calldecodeEntities()on text nodes and on attribute values.source/xml/parseXmlStream.test.js— new tests: decimal/hex numeric references, the five predefined XML entities, pass-through of entity-free text, attribute values.
Performance
decodeEntities() short-circuits (indexOf('&') === -1), so entity-free values — the overwhelming majority in real files — never touch the regex. On a 25 MB synthetic, attribute-heavy sheet (1M attributes): ~180 ms vs ~170 ms parse time (≈5% on that worst-case profile). Typical files show no measurable difference.
Tests
npm test: no new failures. (The 2 failures in parseSheetData.test.js on my machine are pre-existing on master and timezone-related — Europe/Paris DST; unrelated to this change.)