Design reflection over modules

In hackable mode (#365 (closed)), we will want a number of facilities available at run-time:

  1. set! any binding
  2. Use environment to load bindings from any module (at least, those that are part of the compilation unit)
  3. Evaluate an expression within the lexical context of a module

This all relies on a run-time representation of the module tree, which for release builds we carefully avoid building (for inlining and tree-shaking reasons). But now we need to design such a thing, enabled for --hackable builds.

Firstly, for point (1), the answer is that when we register bindings with the module system, we do so wrapped in a case-lambda, like (case-lambda (() foo) ((new-foo) (set! foo new-foo))). The introduced set! will cause the compiler to not be able to see through the binding of foo and so will prohibit most optimizations. In a kind of -O1 build, we may leave off the sets, to allow the compiler to have a little inlining, as a treat.

To actually add the bindings to a module, we need to end up calling module-add! or something, that would add the case-lambda to the module. (Right now there is only module-define! which adds the case-lambda itself.) However there are boot considerations:

  • There are special cases in the expander for (hoot core-syntax) and (hoot core-syntax-helpers), as these need definitions from (hoot expander) but that is much farther down the module loading order; instead these are initialized lazily from the expander itself. See https://gitlab.com/spritely/guile-hoot/-/blob/main/lib/hoot/expander.scm?ref_type=heads#L3260.
  • There is a special case for (hoot primitives), for which most of the primitives need to be eta-expanded into primcalls. See https://gitlab.com/spritely/guile-hoot/-/blob/main/lib/hoot/primitives-module.scm.
  • The reflective module representation is itself a module, and one that is relatively late in the boot order. Either we build the reflective module tree after all modules are loaded, or after (hoot modules) has loaded (and thus current-module becomes available), or we stash reflective definitions as modules are expanded into some kind of boot data structure which is only later unpacked into a module tree.
  • current-module (and environment, and eval, etc) are available to modules that use (hoot modules), so we should be able to reflect on modules from within the compilation unit, not just at the end.

Thinking about this all, I think a workable plan can be:

  • expand-library-group gains some callbacks, before-library, on-definition, after-library, and before-program, which can add statements to the residualized code. These would define modules and add definitions. Before (hoot modules) is seen (and closed), we would not residualize code, and instead defer those initializations until modules are loaded.
  • Perhaps we should add a cond-expand feature for hackable mode, and probably make environment (and similar reflection procedures) error out if not hackable.
  • current-module becomes usable after (hoot modules) has loaded, which is fine. Instead of itself having to make a fake module, it can just load the module from the registry.