Commit 1e2f9c67 authored by Jeronimo Pellegrini's avatar Jeronimo Pellegrini

initial release on hithub

Spartns -- a Common Lisp library for representing sparse matrices and
Spartns is licensed under the terms of the Lisp Lesser GNU
Public License (, known as
the LLGPL. The LLGPL consists of a preamble (see above URL) and the
LGPL. Where these conflict, the preamble takes precedence.
Spartns is referenced in the preamble as the "LIBRARY."
This diff is collapsed.
This diff is collapsed.
Preamble to the Gnu Lesser General Public License
Copyright (c) 2000 Franz Incorporated, Berkeley, CA 94704 (this copytight line refers to the preamble, not to the software)
The concept of the GNU Lesser General Public License version 2.1 ("LGPL") has been adopted to govern the use and distribution of above-mentioned application. However, the LGPL uses terminology that is more appropriate for a program written in C than one written in Lisp. Nevertheless, the LGPL can still be applied to a Lisp program if certain clarifications are made. This document details those clarifications. Accordingly, the license for the open-source Lisp applications consists of this document plus the LGPL. Wherever there is a conflict between this document and the LGPL, this document takes precedence over the LGPL.
A "Library" in Lisp is a collection of Lisp functions, data and foreign modules. The form of the Library can be Lisp source code (for processing by an interpreter) or object code (usually the result of compilation of source code or built with some other mechanisms). Foreign modules are object code in a form that can be linked into a Lisp executable. When we speak of functions we do so in the most general way to include, in addition, methods and unnamed functions. Lisp "data" is also a general term that includes the data structures resulting from defining Lisp classes. A Lisp application may include the same set of Lisp objects as does a Library, but this does not mean that the application is necessarily a "work based on the Library" it contains.
The Library consists of everything in the distribution file set before any modifications are made to the files. If any of the functions or classes in the Library are redefined in other files, then those redefinitions ARE considered a work based on the Library. If additional methods are added to generic functions in the Library, those additional methods are NOT considered a work based on the Library. If Library classes are subclassed, these subclasses are NOT considered a work based on the Library. If the Library is modified to explicitly call other functions that are neither part of Lisp itself nor an available add-on module to Lisp, then the functions called by the modified Library ARE considered a work based on the Library. The goal is to ensure that the Library will compile and run without getting undefined function errors.
It is permitted to add proprietary source code to the Library, but it must be done in a way such that the Library will still run without that proprietary code present. Section 5 of the LGPL distinguishes between the case of a library being dynamically linked at runtime and one being statically linked at build time. Section 5 of the LGPL states that the former results in an executable that is a "work that uses the Library." Section 5 of the LGPL states that the latter results in one that is a "derivative of the Library", which is therefore covered by the LGPL. Since Lisp only offers one choice, which is to link the Library into an executable at build time, we declare that, for the purpose applying the LGPL to the Library, an executable that results from linking a "work that uses the Library" with the Library is considered a "work that uses the Library" and is therefore NOT covered by the LGPL.
Because of this declaration, section 6 of LGPL is not applicable to the Library. However, in connection with each distribution of this executable, you must also deliver, in accordance with the terms and conditions of the LGPL, the source code of Library (or your derivative thereof) that is incorporated into this executable.
The script runs tests across implementations.
You can add more implementations there if you want.
Currently recommended:
The fastest platform for Spartns seems to be SBCL, although
all others seem to run fine (but do run the "do-benchmarks.lisp"
file on your implementation before deciding to use it -- some
implementations are very slow for some representation schemes).
Current state:
1.4.3 1.4.4
ABCL works works
CLISP works works
Clozure works ?
CMUCL works ?
ECL works works
GCL works with warnings
LispWorks works ?
Poplog works ?
SBCL works works
XCL works works
(*) Scieneer passes tests, but something is broken in benchmarks
- For some reason doing (aref M 2) (or any other number) in
Clozure CL doesn't seem to work, so I had to do (coerce 2 'fixnum).
That probably has an impact on performance.
This only afects the ARRAY scheme on Cozure, and only the GET
function (SET and traversals are not affected)
- Poplog does not accept (make-hash-table :size 0), so Spartns
uses #+poplog (max 1 size) #-poplog size
- In Allegro Common Lisp, if A is a SIMPLE-ARRAY, the result of
(adjust-array A new-size) is not.
Spartns now uses a function called ROBUST-ADJUST-ARRAY for this.
ACL is treated as a special case.
- As a consequence of adjusting non-adjustable arrays (as mentioned
in the previous points), growing a CVECTOR beyond its capacity
can be fast or expensive, depending on the Common Lisp implementation
(if your Lisp makes a new array and copies everything over, it will
be O(n); if it is smart enough to optimize this then it can be done in
# Spartns
Spartns is a SPARse TeNSor representation library (if you don't know what a tensor is, think of it as a matrix with any number of dimensions, not just two). Spartns is distributed under the LLGPL license.
* No external dependencies (no BLAS or any other C/Fortran library needed). Just plain Common Lisp;
* Represents mappings from one dimension onto another using any scheme you want (there are three built-in schemes: array, hash and compressed-vector, but you can roll your own and plug it);
* Flexible: works with any data type;
* Heavily optimized: traversing the tensor can be extremely fast (in one specific situation -- traversing the tensor -- it was 10 times faster than a naive implementation in C++);
* Fairly portable: works with SBCL, ABCL, OpenMCL, CMUCL, Clisp, ECL, GCL, XCL, Poplog, LispWorks, and Allegro Common Lisp. But it looks like Spartns does not work with Corman Common Lisp;
* Spartns is never released without going through regression tests (if a platform breaks and can't be supported, it will be clear in the release announcement);
* ASDF installable (thanks Slobodan Blazeski!);
* Easy to use, with introductory documentation (not only on-line);
* Comes with description of the internals of the library.
One simple example: two dimensional matrix multiplication. First, define a type for 2-dimensional matrices:
(defspartn 2dmatrix
:representation (spartns:hash spartns:cvector)
:non-zero (3 4)
:element-type long-float
:sparse-element 0L0)
The Spartn type "2dmatrix" is then defined as the type for sparse tensors that map indices onto long-floats using a hashtable of compressed vectors. When they are created, the hashtables start with :size 3, and the compressed vectors with :size 4. Now, create three matrices, X Y and Z, and multiply them:
(let ((X (make-2dmatrix))
(Y (make-2dmatrix))
(Z (make-2dmatrix)))
(set-2dmatrix X 0 0 5L0)
(set-2dmatrix Y 0 1 6L4)
;; set non-zeros in the rest of the matrices X and Y
;; and now multiply them:
(w/fast-traversals ((2dmatrix X i j val-x)
(2dmatrix Y j k val-y))
(set-2dmatrix Z i k
(+ (get-2dmatrix Z i k) (* val-x val-y)))))
- Write a library of functions that do operations on tensors.
- Add a :include-sparse option to representation scheme traversals
(and also to spartn traversals). We'll need to define total dimensions
besides the number of non-zeros
- Check spartn compatibility. For example, get-hcd can't be used with a
ccd spartn. We should optionally check for the representation scheme
- Optionally check bounds? We actually don't even store them currently.
- Let MAKE-* and some other functions accept a :sparsity-structure
- Include a default sparse-element that makes sense for most Lisp
data types
- The HASH scheme could probably be optimized
- Spartns should optionally generate inline declarations for GET and SET
- Be more defensive and check types in non-speed-critical parts of the code
- Produce HTML from documentation strings
- Expand manual
Possible new representation schemes:
- Particia trees for each dimension?
- AVL trees or skip-lists?
- Chained hash: instead of (gethash i (gethash j data)), we can use
(gethash `(,i ,j) data). This could be interleaved with other schemes
if we're careful:
'(hash cvector chained-hash chained-hash chained-hash array)
would produce:
But the keys to the second hashtable would be lists of three indices:
(get a b c d e f)
(gethash a
(get-cvec (gethash (list c d e) b
(aref data f))))
+ We could use the notation
(hash cvector (hash 3) array)
- Switch from toy tests to something more systematic (the tests should be
+ List all possible problems for each function, and add test cases for them
+ Add more use cases
- Start doing code coverage analysis
- Missing tests:
+ test with ":test-equal EQUAL"
+ generate random elements, insert and get
+ try to use the wrong element type
+ Check if PACK is really working for several dimensions. (Currently the
tests verify that nothing breaks by using it, but we can't tell if it
really is shrinking all intermediate dimensions)
;; This software is Copyright (c) Jeronimo Pellegrini, 2008.
;; You have the rights to distribute
;; and use this software as governed by the terms
;; of the Lisp Lesser GNU Public License
;; (,
;; known as the LLGPL.
;(load "spartns.lisp")
;(compile-file "spartns.lisp")
(in-package :spartns)
#+poplog (setf pop11::popminmemlim 70000000)
; pop11::popgctrace 1)
#+poplog (pop11::sysgarbage)
(defun string->symbol (string &rest args)
(intern (string-upcase (apply #'format `(nil ,string ,@args)))))
(defmacro define-tests (scheme-list)
(let ((result (loop
for scheme in scheme-list collect
`(eval-when (:compile-toplevel :load-toplevel :execute)
,(let ((rep-scheme-name (string->symbol "~a" scheme)))
`(defspartn ,(string->symbol "3d-~a" scheme)
:representation (,rep-scheme-name ,rep-scheme-name ,rep-scheme-name)
:non-zero (100 100 100)
:element-type single-float
:sparse-element 0.0
:def NIL
:declare (optimize (speed 3) (safety 0))))
(defun ,(string->symbol "run-~a-get" scheme) ()
(declare (optimize (speed 3) (safety 0)))
(w/spartns (,(string->symbol "3d-~a" scheme))
(flet ((xloop (the-array)
(let ((value 8.5))
(declare (dynamic-extent value))
(declare (type single-float value))
(dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(setf value (,(string->symbol "get-3d-~a" scheme) the-array i j k)))))))))
(let ((A ( ,(string->symbol "make-3d-~a" scheme) )))
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(,(string->symbol "set-3d-~a" scheme) A i j k -2.0))))
(time (funcall #'xloop A))))))
#-scl (compile ',(string->symbol "run-~a-get" scheme))
(defun ,(string->symbol "run-~a-set" scheme) ()
(declare (optimize (speed 3) (safety 0)))
(w/spartns (,(string->symbol "3d-~a" scheme))
(flet ((xloop (the-array)
(let ((value 8.5))
;(declare (dynamic-extent value))
(declare (type single-float value))
(dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(,(string->symbol "set-3d-~a" scheme) the-array i j k value))))))))
(let ((A ( ,(string->symbol "make-3d-~a" scheme) )))
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(,(string->symbol "set-3d-~a" scheme) A i j k -2.0))))
(time (funcall #'xloop A))))))
#-scl (compile ',(string->symbol "run-~a-set" scheme))
(defun ,(string->symbol "traverse-~a-get" scheme) ()
(declare (optimize (speed 3) (safety 0)))
(w/spartns (,(string->symbol "3d-~a" scheme))
(flet ((xloop (the-array)
(let ((value (the single-float 8.5)))
(declare (dynamic-extent value))
(declare (type single-float value))
(dotimes (n 50)
(,(string->symbol "traverse-3d-~a" scheme) ((i j k) val the-array)
(setf value val))))))
(let ((A ( ,(string->symbol "make-3d-~a" scheme) )))
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(,(string->symbol "set-3d-~a" scheme) A i j k 2.0))))
(time (funcall #'xloop A))))))
#-scl (compile ',(string->symbol "traverse-~a-get" scheme))
(defun ,(string->symbol "traverse-~a-set" scheme) ()
(declare (optimize (speed 3) (safety 0)))
(w/spartns (,(string->symbol "3d-~a" scheme))
(flet ((xloop (the-array)
(let ((value (the single-float 8.5)))
(declare (dynamic-extent value))
(declare (type single-float value))
(dotimes (n 50)
(,(string->symbol "traverse-3d-~a" scheme) ((i j k) val the-array)
(setf val value))))))
(let ((A ( ,(string->symbol "make-3d-~a" scheme) )))
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(,(string->symbol "set-3d-~a" scheme) A i j k 2.0))))
(time (funcall #'xloop A))))))
#-scl (compile ',(string->symbol "traverse-~a-set" scheme))))))
`(progn ,@result)))
(defun run-array-set-old ()
(declare (optimize (speed 3) (debug 0) (safety 0)))
(let ((A (make-array '(100 100 100) :element-type 'single-float :adjustable nil))
(value 8.5))
(declare (dynamic-extent value))
(time (dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(setf (aref A i j k) value))))))))
(defun run-array-set ()
(declare (optimize (speed 3) (debug 0) (safety 0)))
(flet ((xloop ()
(let ((A (make-array '(100 100 100) :element-type 'single-float :adjustable nil))
(value 8.5))
(dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(setf (aref A i j k) value))))))))
(time (funcall #'xloop))))
(compile 'run-array-set)
(defun run-array-get ()
(declare (optimize (speed 3) (debug 0) (safety 0)))
(let ((A (make-array '(100 100 100) :element-type 'single-float :adjustable nil))
(value 8.5))
(time (dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(setf value (aref A i j k)))))))))
(defun run-array-get ()
(declare (optimize (speed 3) (debug 0) (safety 0)))
(flet ((xloop ()
(let ((A (make-array '(100 100 100) :element-type 'single-float :adjustable nil))
(value 8.5))
(dotimes (n 50)
(dotimes (i 90)
(dotimes (j 90)
(dotimes (k 90)
(setf value (aref A i j k)))))))))
(time (funcall #'xloop))))
(compile 'run-array-get)
(defmacro run-tests (scheme-list)
(let ((result
(loop for scheme in scheme-list collect
(format t "~a (SET)~%" ',(string->symbol "RUN-~a-SET" scheme))
(,(string->symbol "RUN-~a-SET" scheme))
(format t "~a (GET)~%" ',(string->symbol "RUN-~a-GET" scheme))
(,(string->symbol "RUN-~a-GET" scheme))
(format t "~a (tr-SET)~%" ',(string->symbol "TRAVERSE-~a-SET" scheme))
(,(string->symbol "TRAVERSE-~a-SET" scheme))
(format t "~a (tr-GET)~%" ',(string->symbol "TRAVERSE-~a-GET" scheme))
(,(string->symbol "TRAVERSE-~a-GET" scheme))))))
(format t "Plain array (SET)~%") (run-array-set)
(format t "Plain array (GET)~%") (run-array-get))))
(define-tests (hash cvector))
(compile 'run-cvector-get)
(compile 'run-cvector-set)
(compile 'traverse-cvector-get)
(compile 'traverse-cvector-set)
(compile 'run-hash-get)
(compile 'run-hash-set)
(compile 'traverse-hash-get)
(compile 'traverse-hash-set)
(run-tests (cvector hash))
Changelog for Spartns
Version numbers are in the format A.B.C; the meaning of changes
in each position is:
A: incompatible changes or major new features
B: new features
C: bugfixes and cleanups
(This was not the case until 0.0.3)
1.4.4 - 07 Jun 2016
* Replaced the previous recursive binary search algorithm. It's
now nonrecursive (it seems that some CL implementations really
don't optimize tail calls)
* Added mising docstrings to utils.lisp
* Reworked benchmark.lisp with suggestion from Juanjo of ECL. Also
fixed a non-serious problem (the names of two twin benchmarks
were swapped: traverse-*-set and traverse-*-get)
* Cleaned up utils.lisp
1.4.3 - 09 Oct 2009
* The docstring issue in XCL has been fixed by Peter Graves!
Now all type declarations and optimize settings are used on
all Common Lisps (the #-xcl hack has been removed!).
1.4.2 - 07 Oct 2009
* Cleaned up tests
1.4.1 - 11 Aug 2009
* Fixed robust-adjust-array so it works without the element-type
argument; also fixed some typos in the documentation.
* Include full text of GPL, LGPL and LLGPL, so there is absolutely
zero information missing for users.
1.4.0 - 11 Mar 2009
* Keeping an index fixed now also works with fast-traversals
1.3.0 - 28 Feb 2009
* It is now possible to traverse while keeping indices fixed.
For example,
(setf i 10)
(traverse-3dmatrix ((i j k) val matrix :dont-bind (i))
will traverse the two rightmost dimensions of matrix
(j, k). i will not be bound, so it will be fixed as
* Document the fact that the ARRAY and HASH schemes are
actually also SETFable while being traversed (and
include this as a test case).
* Document the fact that Spartns works with Scieneer Common
Lisp (except for the benchmarks)
1.2.7 - 05 Oct 2008
* When traversing CVECTORs, values are now SETFable. (This
is not true for other schemes). Documented in manual.
1.2.6 - 01 Oct 2008
* Remove debugging function call...
* Expanded and clarified some online documentation strings
1.2.5 - 01 Oct 2008
* Another attempt to fix w/fast-traversals, including better
1.2.4 - 18 Sep 2008
* Fixed bugs in w/fast-traversals (thanks once again to Yves
* Fix example of w/spartn-traversals usage in the manual
* Also did some cosmetic changes in the manual
1.2.3 - 29 Aug 2008
* Fixes off-by-one bug found by Yves Vandriessche in w/fast-traversals
* Some internal cleanup
* Small updates to documentation: mention MCL, plus some
other additions
1.2.2 - 29 Jun 2008
* Removed some assertions that were causing problems
with GCL
1.2.1 - 29 Jun 2008
* Removed old entries from TODO file
* Updated and cleaned up the manual
1.2.0 - 29 Jun 2008
* New feature: besides resize-amount, it's now possible to
pass a resize-function to defspartn (for example, you can
double the size of a CVECTOR each time it gets full. You
would then use:resize-function (lambda (x) (* 2 x))).
So, "resize-amount 10" is the same as
"resize-function (lambda (x) (+ x 10))"
* Fixed bug when non-zero-list was ommited. Thanks a
lot to Yves Vandriessche for sending a detailed bug
* Small changes to online documentation
* Fixed thinko in PDF manual (thanks Yves Vandriessche)
1.1.6 - 07 Apr 2008
* Load lisp-unit before using it
1.1.5 - 07 Apr 2008
* Fix silly bug in ASDF file
* Some small enhancements to the manual
* Document the fact that Spartns does not work with Corman
Common Lisp
1.1.4 - 05 Apr 2008
* Spartns now works with Clozure Common Lisp
* Fixed symbol generation again (and thanks again to Francis
* Don't require lisp-unit and test suites for ordinary use
(reported by Francis Leboutte)
1.1.3 - 01 Apr 2008
* Traversals on the HASH scheme have been optimized
1.1.2 - 31 Mar 2008
* Include spartns.asd (oops!)
1.1.1 - 31 Mar 2008
* Small optimization (using MULTIPLE-VALUE-BIND instead of
LET) makes the CVECTOR scheme much faster on SBCL
* More cleanups
1.1.0 - 31 Mar 2008
* All schemes now cons much less and are faster (the previous
code would build structures without need)
* Small optimizations
* Function names are now created using uppercase for symbols
(GET-SCHEME instead of |GET-scheme|); thanks to Francis
Leboutte for reporting this
* Cleaned up benchmarks a lot
* Cleaned up tests a bit
1.0.1 - 10 Mar 2008
* Spartns again fully works with Allegro Common Lisp
1.0.0 - 14 Feb 2008
* Fast traversals: if you know that the sparsity structure of
your tensors will not change, you can traverse them *very*
fast. EXCEPT for Allegro Common Lisp
* Equality test is now configurable (not just 'EQL, although
'EQL is still the default)
* The CVECTOR scheme has been enhanced: cvectors will automatically
be resized (by a configurable anmount) when they're full.
EXCEPT for Allegro Common Lisp (it will signal an error)
* It is possible to "pack" sparse tensors, releasing unused memory.
This is useful if you add an unknown number of elements to a
tensor represented as CVECTORS, and end up with extra unused
EXCEPT for Allegro Common Lisp
* Fix one-dimensional cvector spartns (the MAKE-* function was
creating 0-dimensional arrays)
* Clarify the meaning of the :non-zeros key argument to defspartn
* Add a section to the manual explaining how to create tensors
with unknown size, make them static and traverse quickly
0.2.0 - 06 Feb 2008
* New feature: you can traverse several tensors, possibly
stipulating that some indices are shared among tensors
(think matrix multiplication, for example)
* Works with XCL now
* Tests are a bit more robust, but still far from good
0.1.3 - 31 Jan 2008
* benchmark.lisp now adjusts maximum memory for Poplog, so all
functions finish sucessfully
* spartn-copy was also broken; it has been fixed
* More tests added
0.1.2 - 31 Jan 2008
* Fix traversals on the HASH representation scheme
* Fix the ARRAY representation scheme
0.1.1 - 30 Jan 2008
* Now the HASH scheme works with Poplog
* Documentation updates
0.1.0 - 30 Jan 2008
* New representation scheme, ARRAY, for dense dimensions
0.0.4 - 29 Jan 2008
* cvector scheme has been slightly optimized. Conses less and
is a bit faster
* Test utils.lisp, not just spartns
* Some internal cleanup and reorganization
0.0.3 - 29 Jan 2008
* Reformatted changelog
* Some internal cleanup
* Tests work with GCL now
0.0.2 - 28 Jan 2008
* ASDF installable
* Made package and tests more robust with eval-when
* Other minor changes
0.0.1 - 27 Jan 2008
* Initial release
(load "spartns-packages.lisp")
(load "utils.lisp")
(load "spartns.lisp")
(load "benchmark.lisp"))
# This script runs tests on all supported platforms
## Associative array (Bash 4 feature):
declare -A implementations=(
["SBCL"]="sbcl --script"
["ECL"]="ecl -load"
["GCL"]="gcl -load"
["ABCL"]="java -jar /home/jeronimo/pkg/lisp/abcl/abcl.jar --batch --load"
["XCL"]="/home/jeronimo/pkg/lisp/xcl/xcl --load"
echo "- - - - -"
for name in "${!implementations[@]}"; do
echo "$name: "
${implementations["$name"]} do-tests.lisp;
echo "- - - - -"
(load "lisp-unit.lisp")
(load "spartns-packages.lisp")
(load "spartns-test-packages.lisp")
(load "utils.lisp")
(load "spartns.lisp")
(load "utils-test.lisp")
(load "tests.lisp")
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
;; This software is Copyright (c) Jeronimo Pellegrini, 2008.
;; You have the rights to distribute
;; and use this software as governed by the terms
;; of the Lisp Lesser GNU Public License
;; (,
;; known as the LLGPL.
(in-package :cl-user)
(defpackage #:jp-utils
(:use #:common-lisp)
(:export #:binary-search
(defpackage #:spartns
(:use #:common-lisp
(:export #:defscheme
; #:avl ; not ready yet
;; This software is Copyright (c) Jeronimo Pellegrini, 2008.
;; You have the rights to distribute
;; and use this software as governed by the terms
;; of the Lisp Lesser GNU Public License
;; (,
;; known as the LLGPL.
(in-package :cl-user)
(defpackage #:jp-utils-test
(:use #:common-lisp
(defpackage #:spartns-test
(:use #:common-lisp
(asdf:defsystem spartns
:name "spartns"
:version "1.4.4"
:maintainer "Jeronimo C. Pellegrini"
:author "Jeronimo C. Pellegrini"
:licence "LLGPL"
:description "SPARse TeNSor representation library"
:serial t
:components ((:file "spartns-packages")
(:file "utils")
(:file "spartns"))
:in-order-to ((test-op (load-op spartns-test)))
:properties ((#:author-email . "")
((#:albert #:output-dir) . "doc/docbook")
((#:albert #:formats) . ("docbook"))
((#:albert #:docbook #:dtd) . "/usr/share/sgml/docbook/dtd/4.5/docbookx.dtd")))
(asdf:defsystem spartns-test
:depends-on ("spartns")
:serial t
:components ((:file "lisp-unit")