Compilation is not reproducible
Compiling a source file several times can produce different binary files. When making GNU Guix packages for ECL-compiled Common Lisp libraries this is a problem.
I found two functions in the code of the compiler that use the current time or random strings to make unique symbols:
- unique-init-name in cmpname.lsp
- safe-mkstemp in cmpmain.lsp
I don't know if there are some other functions that could bring random things in the binary objects in some cases, but by tweaking these two I was able to get byte-identical xx.fas, xx.fasb and xx.o files when compiling some libraries several times.
Quick-and-dirty proof of concept:
From d17b982e43d01b8496d153bb8a583d4f85417478 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Sat, 11 Jan 2020 16:29:28 +0100
Subject: [PATCH] cmp: Make compilation reproducible.
---
src/cmp/cmpmain.lsp | 31 ++++++++-----------------------
src/cmp/cmpname.lsp | 9 +--------
2 files changed, 9 insertions(+), 31 deletions(-)
diff --git a/src/cmp/cmpmain.lsp b/src/cmp/cmpmain.lsp
index 5e946f4c5..c5f1e0891 100755
--- a/src/cmp/cmpmain.lsp
+++ b/src/cmp/cmpmain.lsp
@@ -20,29 +20,14 @@
(in-package "COMPILER")
-(defun safe-mkstemp (template)
- ;; We do several things here. One is to check for success in MKSTEMP,
- ;; the other one is to ensure that the output of this function _always_
- ;; carries a file type -- this solves a problem with filesystems where
- ;; mkstemp may introduce one or more dots in the name causing several
- ;; functions below to ignore parts of the name. Note that this forces
- ;; us to have two files per temp: one with and one without extension.
- (let* ((base (si::mkstemp template)))
- (when base
- (let ((output (make-pathname :name
- (concatenate 'string (pathname-name base)
- (or (pathname-type base) ""))
- :type "tmp"
- :defaults base)))
- (if (and (not (probe-file output)) (si:copy-file base output))
- (setf base (list (truename output) (truename base)))
- (progn (delete-file base) (setf base nil)))))
- (unless base
- (error "Unable to create temporay file~%~
- ~AXXXXXX
-Make sure you have enough free space in disk, check permissions or set~%~
-the environment variable TMPDIR to a different value." template))
- base))
+(let ((counter 0))
+ (defun safe-mkstemp (template)
+ (incf counter)
+ (let* ((base (pathname (format nil "/tmp/~a~6,'0d" template counter)))
+ (output (make-pathname :type "tmp" :defaults base)))
+ (with-open-file (f base :direction :output))
+ (with-open-file (f output :direction :output))
+ (list (truename output) (truename base)))))
(defun compile-file-pathname (name &key (output-file T) (type nil type-supplied-p)
verbose print c-file h-file data-file
diff --git a/src/cmp/cmpname.lsp b/src/cmp/cmpname.lsp
index dbd340918..cc47491b0 100644
--- a/src/cmp/cmpname.lsp
+++ b/src/cmp/cmpname.lsp
@@ -43,16 +43,9 @@ machine."
(path-hash (logxor (ash (sxhash path) 8)
(ash (sxhash (cddr (pathname-directory path))) 16)
(sxhash (pathname-name path))))
- (seconds (get-universal-time))
- (ms (+ (* seconds 1000)
- (mod (floor (* 1000 (get-internal-real-time))
- internal-time-units-per-second)
- 1000)))
(tag (concatenate 'base-string
"_ecl"
- (encode-number-in-name path-hash)
- "_"
- (encode-number-in-name ms))))
+ (encode-number-in-name path-hash))))
tag))
(defun kind->tag (kind)
--
2.24.1
ECL: 16.1.3 OS: GNU/Linux x86-64