Skip to content

Stackoverflow when using macro from concatenated FASC, but not when loading original FASCs individually

TL;DR: Bytecode compiler stack overflows when loading a macro from a concatenated fasl, but not when loading its constituents individually.

To reproduce this issue, first compile closer-mop into a concatenated FASC file like so:

(ext:install-bytecodes-compiler)

(defun our-compile-file (srcfile)
  "Compile the given srcfile into a compilation unit in :out-dir using
  a unique name based on srcfile as the filename which is returned after
  compilation. If :native is true, create an native object file,
  otherwise a byte-compile fasc file is built and immediately loaded."

  (multiple-value-bind (out-truename _warnings-p failure-p)
    (compile-file srcfile :system-p nil :load t
                  :output-file (make-pathname :name (pathname-name srcfile)
                                              :type "fasc")
                  :verbose t :print t)
      (if failure-p (ext:quit 1) (namestring out-truename))))

(let ((srcs (mapcar #'pathname
                    '("/path/to/closer-mop-packages.lisp"
                      "/path/to/closer-mop-shared.lisp"
                      "/path/to/closer-ecl.lisp"))))

  (with-open-file (fasc-stream "closer-mop.fasc" :direction :output)
    (ext:run-program "cat" (mapcar #'our-compile-file srcs)
                     :output fasc-stream)))

Using the bytecode compiler to compile moptilities (by compiling its single file using compile-file) will work if loading the individual three fasc files one by one, but fail with a binding-stack overflow if loading the concatenated fasc file. I've managed to find a smaller reproducer for this:

This works as expected:

(load "closer-mop-packages.fasc")
(load "closer-mop-shared.fasc")
(load "closer-ecl.fasc")

(in-package :closer-common-lisp)

(defgeneric get-class (thing &key error?)
  (:documentation "Returns the class of thing or nil if the class cannot be found. Thing can be a class, an object representing a class or a symbol naming a class. Get-class is like find-class only not as particular.")
  (:method ((thing symbol) &key error?)
           (find-class thing error?))
  (:method ((thing standard-object) &key error?)
           (declare (ignore error?))
           (class-of thing))
  (:method ((thing t) &key error?) 
           (declare (ignore error?))
           (class-of thing))
  (:method ((thing class) &key error?)
           (declare (ignore error?))
           thing))
$ ecl --load defgeneric-working.lisp
;;; Loading "/home/lukas/src/depot/ecl-repro/defgeneric-working.lisp"
;;; Loading "/home/lukas/src/depot/ecl-repro/closer-mop-packages.fasc"
;;; Loading "/home/lukas/src/depot/ecl-repro/closer-mop-shared.fasc"
;;; Loading "/home/lukas/src/depot/ecl-repro/closer-ecl.fasc"
ECL (Embeddable Common-Lisp) 21.2.1 (git:a8b1c0da43f89800d09c23a27832d0b4c9dcc1e8)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2013 Juan J. Garcia-Ripoll
Copyright (C) 2018 Daniel Kochmanski
Copyright (C) 2021 Daniel Kochmanski and Marius Gerbershagen
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  
Top level in: #<process TOP-LEVEL 0xec1f80>.
> (closer-common-lisp::get-class "foo")

#<The BUILT-IN-CLASS STRING>

Loading the concatenated fasl will make (byte)compiling the macro fail however:

(load "closer-mop.fasc")

(in-package :closer-common-lisp)

(defgeneric get-class (thing &key error?)
  (:documentation "Returns the class of thing or nil if the class cannot be found. Thing can be a class, an object representing a class or a symbol naming a class. Get-class is like find-class only not as particular.")
  (:method ((thing symbol) &key error?)
           (find-class thing error?))
  (:method ((thing standard-object) &key error?)
           (declare (ignore error?))
           (class-of thing))
  (:method ((thing t) &key error?) 
           (declare (ignore error?))
           (class-of thing))
  (:method ((thing class) &key error?)
           (declare (ignore error?))
           thing))
$ ECL (Embeddable Common-Lisp) 21.2.1 (git:a8b1c0da43f89800d09c23a27832d0b4c9dcc1e8)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2013 Juan J. Garcia-Ripoll
Copyright (C) 2018 Daniel Kochmanski
Copyright (C) 2021 Daniel Kochmanski and Marius Gerbershagen
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  
Top level in: #<process TOP-LEVEL 0xd5bf80>.
> (load "defgeneric-broken.lisp")

;;; Loading "/home/lukas/src/depot/ecl-repro/defgeneric-broken.lisp"
;;; Loading "/home/lukas/src/depot/ecl-repro/closer-mop.fasc"
;;; Warning: Class STANDARD-GENERIC-FUNCTION has been forward referenced.

Condition of type: STACK-OVERFLOW
BINDING-STACK overflow at size 10240. Stack can probably be resized.
Proceed with caution.
Available restarts:

1. (CONTINUE) Extend stack size
2. (RESTART-TOPLEVEL) Go back to Top-Level REPL.

Broken at CLOSER-MOP::MAYBE-REMOVE-INITIAL-METHODS. In: #<process TOP-LEVEL 0xd5bf80>.
 File: #P"/nix/store/nc4m300awk689jy8zy3j8m34caf868am-source/closer-mop-shared.lisp" (Position #1128)

This is the function mentioned in the debugger

     VERSION "21.2.1"
      VCS-ID "a8b1c0da43f89800d09c23a27832d0b4c9dcc1e8"
          OS "Linux"
  OS-VERSION "5.10.52"
MACHINE-TYPE "x86_64"
    FEATURES (:CLOSER-MOP :WALKER :CDR-6 :CDR-1 :CDR-5 :LINUX :FORMATTER :CDR-7 :ECL-WEAK-HASH :LITTLE-ENDIAN :ECL-READ-WRITE-LOCK :LONG-LONG :UINT64-T :UINT32-T :UINT16-T :COMPLEX-FLOAT ...)
Edited by sterni