add alternative lisp-modules implementation

based on 4c4cef812e
This commit is contained in:
Kasper Gałkowski 2022-04-25 23:12:22 +02:00
parent e00fbf7fcc
commit 7ac6f7279d
30 changed files with 68370 additions and 0 deletions

View File

@ -0,0 +1 @@
imported.nix linguist-vendored

View File

@ -0,0 +1,3 @@
result
*.sqlite
*.fasl

View File

@ -0,0 +1,197 @@
## The API
This page documents the Nix API of nix-cl.
## Overview
The core API functions are `build-asdf-system` and
`lispWithPackagesInternal`.
They are considered more low-level that the rest of the API, which
builds on top of them to provide a more convenient interface with sane
defaults.
The higher-level API provides a lot of pre-configured packages,
including all of Quicklisp, and consists of the functions:
- `lispPackagesFor`
- `lispWithPackages`
Finally, there are functions that provide pre-defined Lisps, for
people who don't need to customize that:
- `abclPackages`, `eclPackages`, `cclPackages`, `claspPackages`, `sbclPackages`
- `abclWithPackages`, `eclWithPackages`, `cclWithPackages`, `claspWithPackages`, `sbclWithPackages`
The following is an attempt to document all of this.
## Packaging systems - `build-asdf-system`
Packages are declared using `build-asdf-system`. This function takes
the following arguments and returns a `derivation`.
#### Required arguments
##### `pname`
Name of the package/library
##### `version`
Version of the package/library
##### `src`
Source of the package/library (`fetchzip`, `fetchgit`, `fetchhg` etc.)
##### `lisp`
This command must load the provided file (`$buildScript`) then exit
immediately. For example, SBCL's --script flag does just that.
#### Optional arguments
##### `patches ? []`
Patches to apply to the source code before compiling it. This is a
list of files.
##### `nativeLibs ? []`
Native libraries, will be appended to the library
path. (`pkgs.openssl` etc.)
##### `javaLibs ? []`
Java libraries for ABCL, will be appended to the class path.
##### `lispLibs ? []`
Lisp dependencies These must themselves be packages built with
`build-asdf-system`
##### `systems ? [ pname ]`
Some libraries have multiple systems under one project, for example,
[cffi] has `cffi-grovel`, `cffi-toolchain` etc. By default, only the
`pname` system is build.
`.asd's` not listed in `systems` are removed before saving the library
to the Nix store. This prevents ASDF from referring to uncompiled
systems on run time.
Also useful when the `pname` is differrent than the system name, such
as when using [reverse domain naming]. (see `jzon` ->
`com.inuoe.jzon`)
[cffi]: https://cffi.common-lisp.dev/
[reverse domain naming]: https://en.wikipedia.org/wiki/Reverse_domain_name_notation
##### `asds ? systems`
The .asd files that this package provides. By default, same as
`systems`.
#### Return value
A `derivation` that, when built, contains the sources and pre-compiled
FASL files (Lisp implementation dependent) alongside any other
artifacts generated during compilation.
#### Example
[bordeaux-threads.nix] contains a simple example of packaging
`alexandria` and `bordeaux-threads`.
[bordeaux-threads.nix]: /examples/bordeaux-threads.nix
## Building a Lisp with packages: `lispWithPackagesInternal`
Generators of Lisps configured to be able to `asdf:load-system`
pre-compiled libraries on run-time are built with
`lispWithPackagesInternal`.
#### Required Arguments
##### `clpkgs`
An attribute set of `derivation`s returned by `build-asdf-system`
#### Return value
`lispWithPackagesInternal` returns a function that takes one argument:
a function `(lambda (clpkgs) packages)`, that, given a set of
packages, returns a list of package `derivation`s to be included in
the closure.
#### Example
The [sbcl-with-bt.nix] example creates a runnable Lisp where the
`bordeaux-threads` defined in the previous section is precompiled and
loadable via `asdf:load-system`:
[sbcl-with-bt.nix]: /examples/sbcl-with-bt.nix
## Reusing pre-packaged Lisp libraries: `lispPackagesFor`
`lispPackagesFor` is a higher level version of
`lispPackagesForInternal`: it only takes one argument - a Lisp command
to use for compiling packages. It then provides a bunch of ready to
use packages.
#### Required Arguments
##### `lisp`
The Lisp command to use in calls to `build-asdf-system` while building
the library-provided Lisp package declarations.
#### Return value
A set of packages built with `build-asdf-system`.
#### Example
The [abcl-package-set.nix] example generates a set of thousands of packages for ABCL.
[abcl-package-set.nix]: /examples/abcl-package-set.nix
## Reusing pre-packaged Lisp libraries, part 2: `lispWithPackages`
This is simply a helper function to avoid having to call
`lispPackagesFor` if all you want is a Lisp-with-packages wrapper.
#### Required Arguments
##### `lisp`
The Lisp command to pass to `lispPackagesFor` in order for it to
generate a package set. That set is then passed to
`lispWithPackagesInternal`.
#### Return value
A Lisp-with-packages function (see sections above).
#### Example
The [abcl-with-packages.nix] example creates an `abclWithPackages` function.
[abcl-with-packages.nix]: /examples/abcl-with-packages.nix
## Using the default Lisp implementations
This is the easiest way to get going with `nix-cl` in general. Choose
the CL implementation of interest and a set of libraries, and get a
lisp-with-packages wrapper with those libraries pre-compiled.
#### `abclPackages`, `eclPackages`, `cclPackages`, `claspPackages`, `sbclPackages`
Ready to use package sets.
#### `abclWithPackages`, `eclWithPackages`, `cclWithPackages`, `claspWithPackages`, `sbclWithPackages`
Ready to use wrapper generators.
#### Example
For example, to open a shell with SBCL + hunchentoot + sqlite in PATH:
```
nix-shell -p 'with import ./. {}; sbclWithPackages (ps: [ ps.hunchentoot ps.sqlite ])'
```

View File

@ -0,0 +1,98 @@
## Use cases
This page lists some possible use cases for nix-cl.
## Pinning down the exact commits of libraries
Sometimes, a bug is fixed upstream but is not yet available in package
repositories such as Quicklisp or Ultralisp. The users have to wait
for the repository maintainer to update it, or download and compile
the patched sources themselves.
This is a manual and hard to reproduce process. By leveraging Nix,
users of `nix-cl` can essentially "run their own package repository",
written as Nix code, with all the benefits of that (shareability,
cacheability, reproducibility, version-controllable etc.)
## Modifying libraries with patches
Other times, a bug in a library is not fixed upstream, but you fixed
it yourself. Or, you would like a change to the internals that the
maintainers don't like.
Sure, you could fork the code or maintain patches manually, but that
becomes hard to manage with a lot of patches. It also doesn't have the
benefits mentioned in the previous section.
`nix-cl` provides a way of applying version-controlled patches to any
package.
## Using libraries not available in repositories
There are useful and working libraries out there, that are nontheless
unavailable to users of package managers such as Quicklisp or
Ultralisp. Two real-world examples are [jzon] and [cl-tar].
`nix-cl` is not tied to any particular package source: instead,
packages are written as a Nix expression, which can be done manually
or generated/imported.
This frees the user to have any package they want, and not be
constrained by a central repository.
## Reproducible environments
The usual way to develop a project involves several steps, such as:
1. Installing a Lisp implementation
2. Installing a package manager
3. Installing the chosen libraries
This is not necessarily reproducible. It's unlikely to come back a
year later and develop the project using the exact same versions of
the dependencies.
Things can break between attempts at different points in time. The
repository could have updated versions in the meantime. The source
tarballs could become unreachable.
With `nix-cl` you can have your own binary cache for Lisp libraries
and not be affected by downtime of other central repositories.
## Testing across CL implementations
One can manually download different Lisp implementations and run tests
of a package. This works well in most cases, but it is limited in how
you can tweak the software. Some practical examples are:
- Statically compiling [zlib] into [SBCL]
- Building SBCL with the `--fancy` flag
- Compiling [ECL] as a static library
These are usually hard to do manually, unless you have the necessary
compilers already configured. These combinations are usually not
available from package managers as well.
With Nix it's easier, because it will set up the build environment
automatically. It could be useful to, for example:
- Test against all possible compiler flag combinations
- Libc versions (ECL)
- JDK versions ([ABCL])
[zlib]: https://zlib.net
[SBCL]: https://sbcl.org
[ECL]: https://ecl.common-lisp.dev/
[Ultralisp]: https://ultralisp.org/
[jzon]: https://github.com/Zulu-Inuoe/jzon
[cl-tar]: https://gitlab.common-lisp.net/cl-tar/cl-tar
[bootstrap tools]: https://github.com/NixOS/nixpkgs/tree/master/pkgs/stdenv/linux/bootstrap-files
[nixpkgs]: https://github.com/NixOS/nixpkgs
## Windows note
Note that all of this still only applies to Unix systems - primarily because Nix doesn't work on Windows.
If you have an idea how to port some of the functionality to Windows, get in touch.

View File

@ -0,0 +1,54 @@
## Importing package definitions from Quicklisp
This page documents how to import packages from Quicklisp.
## Nix dumper
Run:
```
$ nix-shell
$ sbcl --script ql-import.lisp
```
This command runs a program that dumps a `imported.nix` file
containing Nix expressions for all packages in Quicklisp. They will be
automatically picked up by the `lispPackagesFor` and
`lispWithPackages` API functions.
It also creates a 'packages.sqlite' file. It's used during the
generation of the 'imported.nix' file and can be safely removed. It
contains the full information of Quicklisp packages, so you can use it
to query the dependency graphs using SQL, if you're interested.
## Tarball hashes
The Nix dumper program will re-use hashes from "imported.nix" if it
detects that it's being run for the first time. This saves a lot of
bandwidth by not having to download each tarball again.
But when upgrading the Quicklisp release URL, this can take a while
because it needs to fetch the source code of each new system to
compute its SHA256 hash. This is because Quicklisp only provides a
SHA1 , and Nix's `builtins.fetchTarball` requires a SHA256.
Later on, the hashes are cached in `packages.sqlite`, and are reused
in subsequent invocations. Therefore you might want to keep the
'packages.sqlite' file around if you'd like to keep hashes of
historical Quicklisp tarballs, for example for archival purposes.
## Choosing a Quicklisp release
Quicklisp release url's are currently hard-coded and can be changed
directly in the source code. See the `import` directory.
## Native and Java libraries
At the moment, native and Java libraries need to be added manually to
imported systems in `ql.nix` on an as-needed basis.
## Dependencies from packages.nix
Also worth noting is that systems imported from Quicklisp will prefer
packages from `packages.nix` as dependencies, so that custom versions
can be provided or broken versions replaced.

View File

@ -0,0 +1,5 @@
## Quirks
- `+` in names are converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
- `.` to `_dot_`: `iolib.base`->`iolib_dot_base`
- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)

View File

@ -0,0 +1,24 @@
# To run this example from a nix repl, run:
# $ nix repl
# nix-repl> abcl-packages = import ./abcl-package-set.nix
# nix-repl> builtins.attrNames abcl-packages
# nix-repl> builtins.length (builtins.attrNames abcl-packages)
#
# The import returns a package set, which you can use for example to
# discover what packages are available in lispWithPackages:
#
# nix-repl> abcl-packages.cl-op<TAB>
# nix-repl> abcl-packages.cl-opengl
# nix-repl> # cool, we can use cl-opengl
# nix-repl> # some-abcl-with-packages (p: [ p.cl-opengl ])
let
pkgs = import ../../../../default.nix {};
abcl = "${pkgs.abcl}/bin/abcl --batch --load";
abcl-packages = pkgs.lispPackages_new.lispPackagesFor abcl;
in abcl-packages

View File

@ -0,0 +1,23 @@
# To run this example from a nix repl, run:
# $ nix repl
# nix-repl> abcl-with-packages = import ./abcl-with-packages.nix
# nix-repl> :b abcl-with-packages (p: [ p.cffi ])
#
# The import returns a function, which you can call to get access to
# thousands of libraries, like, cffi. This works in ABCL by closing
# over the JNA dependency:
#
# nix-repl> awp = abcl-with-packages (p: [ p.cffi ])
# nix-repl> awp.CLASSPATH
# nix-repl> cffi = builtins.head (awp.lispLibs)
# nix-repl> cffi.javaLibs
let
pkgs = import ../../../../default.nix {};
abcl = "${pkgs.abcl}/bin/abcl --batch --load";
abcl-with-packages = pkgs.lispPackages_new.lispWithPackages abcl;
in abcl-with-packages

View File

@ -0,0 +1,43 @@
# To run this example from the command line, run this command:
#
# $ nix-build ./bordeaux-threads.nix
# $ ls ./result/
#
# To run from a nix repl, run:
# $ nix repl
# nix-repl> bt = import ./bordeaux-threads.nix
# nix-repl> :b bt
#
# In the `result` directory you can find .fasl files of the
# bordeaux-threads library:
#
# $ ls -l ./result/src/
let
pkgs = import ../../../../default.nix {};
sbcl = "${pkgs.sbcl}/bin/sbcl --script";
alexandria = pkgs.lispPackages_new.build-asdf-system {
pname = "alexandria";
version = "v1.4";
src = pkgs.fetchzip {
url = "https://gitlab.common-lisp.net/alexandria/alexandria/-/archive/v1.4/alexandria-v1.4.tar.gz";
sha256 = "0r1adhvf98h0104vq14q7y99h0hsa8wqwqw92h7ghrjxmsvz2z6l";
};
lisp = sbcl;
};
bordeaux-threads = pkgs.lispPackages_new.build-asdf-system {
pname = "bordeaux-threads";
version = "0.8.8";
src = pkgs.fetchzip {
url = "http://github.com/sionescu/bordeaux-threads/archive/v0.8.8.tar.gz";
sha256 = "19i443fz3488v1pbbr9x24y8h8vlyhny9vj6c9jk5prm702awrp6";
};
lisp = sbcl;
lispLibs = [ alexandria ];
};
in bordeaux-threads

View File

@ -0,0 +1,31 @@
# To run this example from the command line, run this command:
# $ nix-build ./sbcl-with-bt.nix
# $ ls ./result/
#
# To run from a nix repl, run:
# $ nix repl
# nix-repl> sbcl-bt = import ./sbcl-with-bt.nix
# nix-repl> :b sbcl-bt
#
# In the `result/bin` directory you can find an `sbcl` executable
# that, when started, is able to load the pre-compiled
# bordeaux-threads from the Nix store:
# $ ./result/bin/sbcl
# * (require :asdf)
# * (asdf:load-system :bordeaux-threads)
let
pkgs = import ../../../../default.nix {};
sbcl = "${pkgs.sbcl}/bin/sbcl --script";
bordeaux-threads = import ./bordeaux-threads.nix;
sbclPackages = { inherit bordeaux-threads; };
sbclWithPackages = pkgs.lispPackages_new.lispWithPackagesInternal sbclPackages;
sbcl-bt = sbclWithPackages (p: [ p.bordeaux-threads ]);
in sbcl-bt

View File

@ -0,0 +1,18 @@
(defpackage org.lispbuilds.nix/api
(:documentation "Public interface of org.lispbuilds.nix")
(:use :cl)
(:export
:import-lisp-packages
:database->nix-expression))
(in-package org.lispbuilds.nix/api)
(defgeneric import-lisp-packages (repository database)
(:documentation
"Import Lisp packages (ASDF systems) from repository (Quicklisp,
Ultralisp etc.) into a package database."))
(defgeneric database->nix-expression (database outfile)
(:documentation
"Generate a nix expression from the package database and write it
into outfile."))

View File

@ -0,0 +1,134 @@
(defpackage org.lispbuilds.nix/database/sqlite
(:use :cl)
(:import-from :str)
(:import-from :sqlite)
(:import-from :alexandria :read-file-into-string)
(:import-from :arrow-macros :->>)
(:import-from
:org.lispbuilds.nix/util
:replace-regexes)
(:import-from
:org.lispbuilds.nix/nix
:nix-eval
:system-master
:nixify-symbol
:make-pname
:*nix-attrs-depth*)
(:import-from
:org.lispbuilds.nix/api
:database->nix-expression)
(:export :sqlite-database :init-db)
(:local-nicknames
(:json :com.inuoe.jzon)))
(in-package org.lispbuilds.nix/database/sqlite)
(defclass sqlite-database ()
((url :initarg :url
:reader database-url
:initform (error "url required"))
(init-file :initarg :init-file
:reader init-file
:initform (error "init file required"))))
(defun init-db (db init-file)
(let ((statements (->> (read-file-into-string init-file)
(replace-regexes '(".*--.*") '(""))
(substitute #\Space #\Newline)
(str:collapse-whitespaces)
(str:split #\;)
(mapcar #'str:trim)
(remove-if #'str:emptyp))))
(sqlite:with-transaction db
(dolist (s statements)
(sqlite:execute-non-query db s)))))
;; Writing Nix
(defparameter prelude "
# This file was auto-generated by nix-quicklisp.lisp
{ runCommand, fetchzip, pkgs, ... }:
# Ensures that every non-slashy `system` exists in a unique .asd file.
# (Think cl-async-base being declared in cl-async.asd upstream)
#
# This is required because we're building and loading a system called
# `system`, not `asd`, so otherwise `system` would not be loadable
# without building and loading `asd` first.
#
let createAsd = { url, sha256, asd, system }:
let
src = fetchzip { inherit url sha256; };
in runCommand \"source\" {} ''
mkdir -pv $out
cp -r ${src}/* $out
find $out -name \"${asd}.asd\" | while read f; do mv -fv $f $(dirname $f)/${system}.asd || true; done
'';
getAttr = builtins.getAttr;
in {")
(defmethod database->nix-expression ((database sqlite-database) outfile)
(sqlite:with-open-database (db (database-url database))
(with-open-file (f outfile
:direction :output
:if-exists :supersede)
;; Fix known problematic packages before dumping the nix file.
(sqlite:execute-non-query db
"create temp table fixed_systems as select * from system_view")
(sqlite:execute-non-query db
"alter table fixed_systems add column systems")
(sqlite:execute-non-query db
"update fixed_systems set systems = json_array(name)")
(sqlite:execute-non-query db
"alter table fixed_systems add column asds")
(sqlite:execute-non-query db
"update fixed_systems set asds = json_array(name)")
(format f prelude)
(dolist (p (sqlite:execute-to-list db "select * from fixed_systems"))
(destructuring-bind (name version asd url sha256 deps systems asds) p
(format f "~% ")
(let ((*nix-attrs-depth* 1))
(format
f
"~a = ~a;"
(nix-eval `(:symbol ,name))
(nix-eval
`(:attrs
("pname" (:string ,(make-pname name)))
("version" (:string ,version))
("asds" (:list
,@(mapcar (lambda (asd)
`(:string ,(system-master asd)))
(coerce (json:parse asds) 'list))))
("src" (:funcall
"createAsd"
(:attrs
("url" (:string ,url))
("sha256" (:string ,sha256))
("system" (:string ,(system-master name)))
("asd" (:string ,asd)))))
("systems" (:list
,@(mapcar (lambda (sys)
`(:string ,sys))
(coerce (json:parse systems) 'list))))
("lispLibs" (:list
,@(mapcar (lambda (dep)
`(:funcall
"getAttr"
(:string ,(nixify-symbol dep))
(:symbol "pkgs")))
(remove "asdf"
(str:split-omit-nulls #\, deps)
:test #'string=))))))))))
(format f "~%}~%"))))

View File

@ -0,0 +1,41 @@
CREATE TABLE IF NOT EXISTS sha256 (
id integer PRIMARY KEY AUTOINCREMENT,
url text UNIQUE,
hash text NOT NULL,
created real DEFAULT (julianday('now'))
);
CREATE TABLE IF NOT EXISTS system (
id integer PRIMARY KEY AUTOINCREMENT,
name text NOT NULL,
version text NOT NULL,
asd text NOT NULL,
created real DEFAULT (julianday('now')),
UNIQUE(name, version)
);
CREATE TABLE IF NOT EXISTS dep (
system_id integer NOT NULL REFERENCES system(id),
dep_id integer NOT NULL REFERENCES system(id),
PRIMARY KEY (system_id, dep_id)
);
CREATE TABLE IF NOT EXISTS src (
sha256_id integer REFERENCES sha256(id),
system_id integer UNIQUE REFERENCES system(id)
);
DROP VIEW IF EXISTS system_view;
CREATE VIEW IF NOT EXISTS system_view AS
SELECT
sys.name,
sys.version,
sys.asd,
sha.url,
sha.hash,
group_concat((SELECT name FROM system WHERE id = dep.dep_id)) as deps
FROM system sys
JOIN src ON src.system_id = sys.id
JOIN sha256 sha ON sha.id = src.sha256_id
LEFT JOIN dep ON dep.system_id = sys.id
GROUP BY sys.name;

View File

@ -0,0 +1,40 @@
(defpackage org.lispbuilds.nix/main
(:use :common-lisp
:org.lispbuilds.nix/database/sqlite
:org.lispbuilds.nix/repository/quicklisp
:org.lispbuilds.nix/api))
(in-package org.lispbuilds.nix/main)
(defun resource (name type)
(make-pathname
:defaults (asdf:system-source-directory :org.lispbuilds.nix)
:name name
:type type))
(defvar *sqlite*
(make-instance
'sqlite-database
:init-file (resource "init" "sql")
:url "packages.sqlite"))
(defvar *quicklisp*
(make-instance
'quicklisp-repository
:dist-url
"https://beta.quicklisp.org/dist/quicklisp/2021-12-30/"))
(defun run-importers ()
(import-lisp-packages *quicklisp* *sqlite*)
(format t "Imported packages from quicklisp to ~A~%"
(truename "packages.sqlite")))
(defun gen-nix-file ()
(database->nix-expression *sqlite* "imported.nix")
(format t "Dumped nix file to ~a~%"
(truename "imported.nix")))
(defun main ()
(format t "~%")
(run-importers)
(gen-nix-file))

View File

@ -0,0 +1,81 @@
(defpackage org.lispbuilds.nix/nix
(:documentation "Utilities for generating Nix code")
(:use :cl)
(:import-from :str)
(:import-from :ppcre)
(:import-from :arrow-macros :->>)
(:import-from :org.lispbuilds.nix/util :replace-regexes)
(:export
:nix-eval
:system-master
:nixify-symbol
:make-pname
:*nix-attrs-depth*))
(in-package org.lispbuilds.nix/nix)
;; Path names are alphanumeric and can include the symbols +-._?= and
;; must not begin with a period.
(defun make-pname (string)
(replace-regexes '("^[.]" "[^a-zA-Z0-9+-._?=]")
'("_" "_")
string))
(defun system-master (system)
(first (str:split "/" system)))
;;;; Nix generation
(defun nix-eval (exp)
(assert (consp exp))
(ecase (car exp)
(:string (nix-string (cadr exp)))
(:list (apply #'nix-list (rest exp)))
(:funcall (apply #'nix-funcall (rest exp)))
(:attrs (nix-attrs (cdr exp)))
(:merge (apply #'nix-merge (cdr exp)))
(:symbol (nix-symbol (cadr exp)))))
(defun nix-string (object)
(format nil "\"~a\"" object))
(defun nixify-symbol (string)
(flet ((fix-special-chars (str)
(replace-regexes '("[+]$" "[+][/]" "[+]" "[.]" "[/]")
'("_plus" "_plus/" "_plus_" "_dot_" "_slash_")
str)))
(if (ppcre:scan "^[0-9]" string)
(str:concat "_" (fix-special-chars string))
(fix-special-chars string))))
(defun nix-symbol (object)
(nixify-symbol (format nil "~a" object)))
(defun nix-list (&rest things)
(format nil "[ ~{~A~^ ~} ]" (mapcar 'nix-eval things)))
(defvar *nix-attrs-depth* 0)
(defun nix-attrs (keyvals)
(let ((*nix-attrs-depth* (1+ *nix-attrs-depth*)))
(format
nil
(->> "{~%*depth*~{~{~A = ~A;~}~^~%*depth*~}~%*depth-1*}"
(str:replace-all "*depth*" (str:repeat *nix-attrs-depth* " "))
(str:replace-all "*depth-1*" (str:repeat (1- *nix-attrs-depth*) " ")))
(mapcar (lambda (keyval)
(let ((key (car keyval))
(val (cadr keyval)))
(list (nix-symbol key)
(nix-eval val))))
keyvals))))
(defun nix-funcall (fun &rest args)
(format nil "(~a ~{~a~^ ~})"
(nixify-symbol fun)
(mapcar 'nix-eval args)))
(defun nix-merge (a b)
(format nil "(~a // ~b)"
(nix-eval a)
(nix-eval b)))

View File

@ -0,0 +1,24 @@
(defsystem org.lispbuilds.nix
:class :package-inferred-system
:description "Utilities for importing ASDF systems into Nix"
:depends-on (
:alexandria
:str
:cl-ppcre
:sqlite
:dexador
:arrow-macros
:com.inuoe.jzon
:org.lispbuilds.nix/api
:org.lispbuilds.nix/repository/quicklisp
:org.lispbuilds.nix/database/sqlite
))
(register-system-packages
"cl-ppcre"
'(:ppcre))
(register-system-packages
"dexador"
'(:dex))

View File

@ -0,0 +1,199 @@
(defpackage org.lispbuilds.nix/repository/quicklisp
(:use :cl)
(:import-from :dex)
(:import-from :alexandria :read-file-into-string :ensure-list)
(:import-from :arrow-macros :->>)
(:import-from :str)
(:import-from
:org.lispbuilds.nix/database/sqlite
:sqlite-database
:init-db
:database-url
:init-file)
(:import-from
:org.lispbuilds.nix/api
:import-lisp-packages)
(:import-from
:org.lispbuilds.nix/util
:replace-regexes)
(:export :quicklisp-repository)
(:local-nicknames
(:json :com.inuoe.jzon)))
(in-package org.lispbuilds.nix/repository/quicklisp)
(defclass quicklisp-repository ()
((dist-url :initarg :dist-url
:reader dist-url
:initform (error "dist url required"))))
(defun clear-line ()
(write-char #\Return *error-output*)
(write-char #\Escape *error-output*)
(write-char #\[ *error-output*)
(write-char #\K *error-output*))
(defun status (&rest format-args)
(clear-line)
(apply #'format (list* *error-output* format-args))
(force-output *error-output*))
;; TODO: This should not know about the imported.nix file.
(defun init-tarball-hashes (database)
(status "no packages.sqlite - will pre-fill tarball hashes from ~A to save time~%"
(truename "imported.nix"))
(let* ((lines (uiop:read-file-lines "imported.nix"))
(lines (remove-if-not
(lambda (line)
(let ((trimmed (str:trim-left line)))
(or (str:starts-with-p "url = " trimmed)
(str:starts-with-p "sha256 = " trimmed))))
lines))
(lines (mapcar
(lambda (line)
(multiple-value-bind (whole groups)
(ppcre:scan-to-strings "\"\(.*\)\"" line)
(declare (ignore whole))
(svref groups 0)))
lines)))
(sqlite:with-open-database (db (database-url database))
(init-db db (init-file database))
(sqlite:with-transaction db
(loop while lines do
(sqlite:execute-non-query db
"insert or ignore into sha256(url,hash) values (?,?)"
(prog1 (first lines) (setf lines (rest lines)))
(prog1 (first lines) (setf lines (rest lines))))))
(status "OK, imported ~A hashes into DB.~%"
(sqlite:execute-single db
"select count(*) from sha256")))))
(defmethod import-lisp-packages ((repository quicklisp-repository)
(database sqlite-database))
;; If packages.sqlite is missing, we should populate the sha256
;; table to speed things up.
(unless (probe-file (database-url database))
(init-tarball-hashes database))
(let* ((db (sqlite:connect (database-url database)))
(systems-url (str:concat (dist-url repository) "systems.txt"))
(releases-url (str:concat (dist-url repository) "releases.txt"))
(systems-lines (rest (butlast (str:split #\Newline (dex:get systems-url)))))
(releases-lines (rest (butlast (str:split #\Newline (dex:get releases-url))))))
(flet ((sql-query (sql &rest params)
(apply #'sqlite:execute-to-list (list* db sql params))))
;; Ensure database schema
(init-db db (init-file database))
;; Prepare temporary tables for efficient access
(sql-query "create temp table if not exists quicklisp_system
(project, asd, name unique, deps)")
(sql-query "create temp table if not exists quicklisp_release
(project unique, url, size, md5, sha1, prefix not null, asds)")
(sqlite:with-transaction db
(dolist (line systems-lines)
(destructuring-bind (project asd name &rest deps)
(str:words line)
(sql-query
"insert or ignore into quicklisp_system values(?,?,?,?)"
project asd name (json:stringify (coerce deps 'vector))))))
(sqlite:with-transaction db
(dolist (line releases-lines)
(destructuring-bind (project url size md5 sha1 prefix &rest asds)
(str:words line)
(sql-query
"insert or ignore into quicklisp_release values(?,?,?,?,?,?,?)"
project url size md5 sha1 prefix (json:stringify (coerce
asds
'vector))))))
(sqlite:with-transaction db
;; Should these be temp tables, that then get queried by
;; system name? This looks like it uses a lot of memory.
(let ((systems
(sql-query
"with pkg as (
select
name, asd, url, deps,
ltrim(replace(prefix, r.project, ''), '-_') as version
from quicklisp_system s, quicklisp_release r
where s.project = r.project
)
select
name, version, asd, url,
(select json_group_array(
json_array(value, (select version from pkg where name=value))
)
from json_each(deps)) as deps
from pkg"
)))
;; First pass: insert system and source tarball informaton.
;; Can't insert dependency information, because this works
;; on system ids in the database and they don't exist
;; yet. Could it be better to just base dependencies on
;; names? But then ACID is lost.
(dolist (system systems)
(destructuring-bind (name version asd url deps) system
(declare (ignore deps))
(status "importing system '~a-~a'" name version)
(let ((hash (nix-prefetch-tarball url db)))
(sql-query
"insert or ignore into system(name,version,asd) values (?,?,?)"
name version asd)
(sql-query
"insert or ignore into sha256(url,hash) values (?,?)"
url hash)
(sql-query
"insert or ignore into src values
((select id from sha256 where url=?),
(select id from system where name=? and version=?))"
url name version))))
;; Second pass: connect the in-database systems with
;; dependency information
(dolist (system systems)
(destructuring-bind (name version asd url deps) system
(declare (ignore asd url))
(dolist (dep (coerce (json:parse deps) 'list))
(destructuring-bind (dep-name dep-version) (coerce dep 'list)
(if (eql dep-version 'NULL)
(warn "Bad data in Quicklisp: ~a has no version" dep-name)
(sql-query
"insert or ignore into dep values
((select id from system where name=? and version=?),
(select id from system where name=? and version=?))"
name version
dep-name dep-version))))))))))
(write-char #\Newline *error-output*))
(defun shell-command-to-string (cmd)
;; Clearing the library path is needed to prevent a bug, where the
;; called subprocess uses a different glibc than the SBCL process
;; is. In that case, the call to execve attempts to load the
;; libraries used by SBCL from LD_LIBRARY_PATH using a different
;; glibc than they expect, which errors out.
(let ((ld-library-path (uiop:getenv "LD_LIBRARY_PATH")))
(setf (uiop:getenv "LD_LIBRARY_PATH") "")
(unwind-protect
(uiop:run-program cmd :output '(:string :stripped t))
(setf (uiop:getenv "LD_LIBRARY_PATH") ld-library-path))))
(defun nix-prefetch-tarball (url db)
(restart-case
(compute-sha256 url db)
(try-again ()
:report "Try downloading again"
(nix-prefetch-tarball url db))))
(defun compute-sha256 (url db)
(or (sqlite:execute-single db "select hash from sha256 where url=?" url)
(let ((sha256 (shell-command-to-string (str:concat "nix-prefetch-url --unpack " url))))
sha256)))

View File

@ -0,0 +1,16 @@
(defpackage org.lispbuilds.nix/util
(:use :cl)
(:import-from :ppcre)
(:export
:replace-regexes))
(in-package org.lispbuilds.nix/util)
(defun replace-regexes (from to str)
(assert (= (length from) (length to)))
(if (null from)
str
(replace-regexes
(rest from)
(rest to)
(ppcre:regex-replace-all (first from) str (first to)))))

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,432 @@
# TODO:
# - faster build by using lisp with preloaded asdf?
# - dont include java libs unless abcl?
# - dont use build-asdf-system to build lispWithPackages?
# - make the lisp packages overridable? (e.g. buildInputs glibc->musl)
# - build asdf with nix and use that instead of one shipped with impls
# (e.g. to fix build with clisp - does anyone use clisp?)
# - claspPackages ? (gotta package clasp with nix first)
# - hard one: remove unrelated sources ( of systems not being built)
# - figure out a less awkward way to patch sources
# (have to build from src directly for SLIME to work, so can't just patch sources in place)
{ pkgs, lib, stdenv, ... }:
let
inherit (lib)
length
filter
foldl
unique
id
concat
concatMap
mutuallyExclusive
findFirst
remove
setAttr
getAttr
hasAttr
attrNames
attrValues
filterAttrs
mapAttrs
splitString
concatStringsSep
concatMapStringsSep
replaceStrings
removeSuffix
hasInfix
optionalString
makeLibraryPath
makeSearchPath
;
inherit (builtins)
head
tail
elem
split
storeDir;
# Returns a flattened dependency tree without duplicates
# This is probably causing performance problems...
flattenedDeps = lispLibs:
let
walk = acc: node:
if length node.lispLibs == 0
then acc
else foldl walk (acc ++ node.lispLibs) node.lispLibs;
in unique (walk [] { inherit lispLibs; });
# Stolen from python-packages.nix
# Actually no idea how this works
makeOverridableLispPackage = f: origArgs:
let
ff = f origArgs;
overrideWith = newArgs: origArgs // (if pkgs.lib.isFunction newArgs then newArgs origArgs else newArgs);
in
if builtins.isAttrs ff then (ff // {
overrideLispAttrs = newArgs: makeOverridableLispPackage f (overrideWith newArgs);
})
else if builtins.isFunction ff then {
overrideLispAttrs = newArgs: makeOverridableLispPackage f (overrideWith newArgs);
__functor = self: ff;
}
else ff;
#
# Wrapper around stdenv.mkDerivation for building ASDF systems.
#
build-asdf-system = makeOverridableLispPackage (
{ pname,
version,
src ? null,
patches ? [],
# Native libraries, will be appended to the library path
nativeLibs ? [],
# Java libraries for ABCL, will be appended to the class path
javaLibs ? [],
# Lisp dependencies
# these should be packages built with `build-asdf-system`
lispLibs ? [],
# Lisp command to run buildScript
lisp,
# Some libraries have multiple systems under one project, for
# example, cffi has cffi-grovel, cffi-toolchain etc. By
# default, only the `pname` system is build.
#
# .asd's not listed in `systems` are removed in
# installPhase. This prevents asdf from referring to uncompiled
# systems on run time.
#
# Also useful when the pname is differrent than the system name,
# such as when using reverse domain naming.
systems ? [ pname ],
# The .asd files that this package provides
asds ? systems,
# Other args to mkDerivation
...
} @ args:
let
# A little slow for big dependency graphs.
# How to make it faster?
# Maybe do it in the buildScript in Common Lisp or Bash, instead
# of here with Nix?
libsFlat = flattenedDeps lispLibs;
in stdenv.mkDerivation (rec {
inherit pname version nativeLibs javaLibs lispLibs lisp systems asds;
src = if builtins.length patches > 0
then apply-patches args
else args.src;
# When src is null, we are building a lispWithPackages and only
# want to make use of the dependency environment variables
# generated by build-asdf-system
dontUnpack = src == null;
# TODO: Do the propagation like for lisp, native and java like this:
# https://github.com/teu5us/nix-lisp-overlay/blob/e30dafafa5c1b9a5b0ccc9aaaef9d285d9f0c46b/pkgs/development/lisp-modules/setup-hook.sh
# Then remove the "echo >> nix-drvs" from buildScript
# Tell asdf where to find system definitions of lisp dependencies.
#
# The "//" ending is important as it makes asdf recurse into
# subdirectories when searching for .asd's. This is to support
# projects where .asd's aren't in the root directory.
CL_SOURCE_REGISTRY = makeSearchPath "/" libsFlat;
# Tell lisp where to find native dependencies
#
# Normally generated from lispLibs, but LD_LIBRARY_PATH as a
# derivation attr itself can be used as an extension point when
# the libs are not in a '/lib' subdirectory
LD_LIBRARY_PATH =
let
libs = concatMap (x: x.nativeLibs) libsFlat;
paths = filter (x: x != "") (map (x: x.LD_LIBRARY_PATH) libsFlat);
path =
makeLibraryPath libs
+ optionalString (length paths != 0) ":"
+ concatStringsSep ":" paths;
in concatStringsSep ":" (unique (splitString ":" path));
# Java libraries For ABCL
CLASSPATH = makeSearchPath "share/java/*" (concatMap (x: x.javaLibs) libsFlat);
# Portable script to build the systems.
#
# `lisp` must evaluate this file then exit immediately. For
# example, SBCL's --script flag does just that.
#
# NOTE:
# Every other library worked fine with asdf:compile-system in
# buildScript.
#
# cl-syslog, for some reason, signals that CL-SYSLOG::VALID-SD-ID-P
# is undefined with compile-system, but works perfectly with
# load-system. Strange.
buildScript = pkgs.writeText "build-${pname}.lisp" ''
(require :asdf)
(dolist (s '(${concatStringsSep " " systems}))
(asdf:load-system s))
'';
buildPhase = optionalString (src != null) ''
# In addition to lisp dependencies, make asdf see the .asd's
# of the systems being built
#
# *Append* src since `lispLibs` can provide .asd's that are
# also in `src` but are not in `systems` (that is, the .asd's
# that will be deleted in installPhase). We don't want to
# rebuild them, but to load them from lispLibs.
#
# NOTE: It's important to read files from `src` instead of
# from pwd to get go-to-definition working with SLIME
export CL_SOURCE_REGISTRY=$CL_SOURCE_REGISTRY:${src}//
# Similiarily for native deps
export LD_LIBRARY_PATH=${makeLibraryPath nativeLibs}:$LD_LIBRARY_PATH
export CLASSPATH=${makeSearchPath "share/java/*" javaLibs}:$CLASSPATH
# Make asdf compile from `src` to pwd and load `lispLibs`
# from storeDir. Otherwise it could try to recompile lisp deps.
export ASDF_OUTPUT_TRANSLATIONS="${src}:$(pwd):${storeDir}:${storeDir}"
# Make Nix track the dependencies so that graphs can be generated with
# nix-store -q --graph
echo "$lispLibs" >> nix-drvs
echo "$nativeLibs" >> nix-drvs
echo "$javaLibs" >> nix-drvs
# Finally, compile the systems
${lisp} $buildScript
'';
# Copy compiled files to store
#
# Make sure to include '$' in regex to prevent skipping
# stuff like 'iolib.asdf.asd' for system 'iolib.asd'
#
# Same with '/': `local-time.asd` for system `cl-postgres+local-time.asd`
installPhase =
let
mkSystemsRegex = systems:
concatMapStringsSep "\\|" (replaceStrings ["." "+"] ["[.]" "[+]"]) systems;
in
''
mkdir -pv $out
cp -r * $out
# Remove all .asd files except for those in `systems`.
find $out -name "*.asd" \
| grep -v "/\(${mkSystemsRegex systems}\)\.asd$" \
| xargs rm -fv || true
'';
# Not sure if it's needed, but caused problems with SBCL
# save-lisp-and-die binaries in the past
dontStrip = true;
dontFixup = true;
} // args));
# Need to do that because we always want to compile straight from
# `src` for go-to-definition to work in SLIME.
apply-patches = { patches, src, ... }:
stdenv.mkDerivation {
inherit patches src;
pname = "source";
version = "patched";
dontConfigure = true;
dontBuild = true;
dontStrip = true;
dontFixup = true;
installPhase = ''
mkdir -pv $out
cp -r * $out
'';
};
# Build the set of lisp packages using `lisp`
# These packages are defined manually for one reason or another:
# - The library is not in quicklisp
# - The library that is in quicklisp is broken
# - Special build procedure such as cl-unicode, asdf
#
# These Probably could be done even in ql.nix
# - Want to pin a specific commit
# - Want to apply custom patches
#
# They can use the auto-imported quicklisp packages as dependencies,
# but some of those don't work out of the box.
#
# E.g if a QL package depends on cl-unicode it won't build out of
# the box. The dependency has to be rewritten using the manually
# fixed cl-unicode.
#
# This is done by generating a 'fixed' set of Quicklisp packages by
# calling quicklispPackagesFor with the right `fixup`.
commonLispPackagesFor = lisp:
let
build-asdf-system' = body: build-asdf-system (body // { inherit lisp; });
in import ./packages.nix {
inherit pkgs;
inherit lisp;
inherit quicklispPackagesFor;
inherit fixupFor;
build-asdf-system = build-asdf-system';
};
# Build the set of packages imported from quicklisp using `lisp`
quicklispPackagesFor = { lisp, fixup ? lib.id, build ? build-asdf-system }:
let
build-asdf-system' = body: build (body // {
inherit lisp;
});
in import ./ql.nix {
inherit pkgs;
inherit fixup;
build-asdf-system = build-asdf-system';
};
# Rewrite deps of pkg to use manually defined packages
#
# The purpose of manual packages is to customize one package, but
# then it has to be propagated everywhere for it to make sense and
# have consistency in the package tree.
fixupFor = manualPackages: qlPkg:
assert (lib.isAttrs qlPkg && !lib.isDerivation qlPkg);
let
# Make it possible to reuse generated attrs without recursing into oblivion
packages = (lib.filterAttrs (n: v: n != qlPkg.pname) manualPackages);
substituteLib = pkg:
if lib.hasAttr pkg.pname packages
then packages.${pkg.pname}
else pkg;
pkg = substituteLib qlPkg;
in pkg // { lispLibs = map substituteLib pkg.lispLibs; };
makeAttrName = str:
removeSuffix
"_"
(replaceStrings
["+" "." "/"]
["_plus_" "_dot_" "_slash_"]
str);
oldMakeWrapper = pkgs.runCommand "make-wrapper.sh" {} ''
substitute ${./old-make-wrapper.sh} $out \
--replace @shell@ ${pkgs.bash}/bin/bash
'';
# Creates a lisp wrapper with `packages` installed
#
# `packages` is a function that takes `clpkgs` - a set of lisp
# packages - as argument and returns the list of packages to be
# installed
lispWithPackagesInternal = clpkgs: packages:
# FIXME just use flattenedDeps instead
(build-asdf-system rec {
lisp = (head (lib.attrValues clpkgs)).lisp;
# See dontUnpack in build-asdf-system
src = null;
pname = baseNameOf (head (split " " lisp));
version = "with-packages";
lispLibs = packages clpkgs;
systems = [];
}).overrideAttrs(o: {
installPhase = ''
# The recent version of makeWrapper causes breakage. For more info see
# https://github.com/Uthar/nix-cl/issues/2
source ${oldMakeWrapper}
mkdir -pv $out/bin
makeWrapper \
${head (split " " o.lisp)} \
$out/bin/${baseNameOf (head (split " " o.lisp))} \
--prefix CL_SOURCE_REGISTRY : "${o.CL_SOURCE_REGISTRY}" \
--prefix ASDF_OUTPUT_TRANSLATIONS : ${concatStringsSep "::" (flattenedDeps o.lispLibs)}: \
--prefix LD_LIBRARY_PATH : "${o.LD_LIBRARY_PATH}" \
--prefix LD_LIBRARY_PATH : "${makeLibraryPath o.nativeLibs}" \
--prefix CLASSPATH : "${o.CLASSPATH}" \
--prefix CLASSPATH : "${makeSearchPath "share/java/*" o.javaLibs}"
'';
});
lispWithPackages = lisp:
let
packages = lispPackagesFor lisp;
in lispWithPackagesInternal packages;
lispPackagesFor = lisp:
let
packages = commonLispPackagesFor lisp;
qlPackages = quicklispPackagesFor {
inherit lisp;
fixup = fixupFor packages;
};
in qlPackages // packages;
commonLispPackages = rec {
inherit
build-asdf-system
lispWithPackagesInternal
lispPackagesFor
lispWithPackages;
# Uncomment for debugging/development
# inherit
# flattenedDeps
# concatMap
# attrNames
# getAttr
# filterAttrs
# filter
# elem
# unique
# makeAttrName
# length;
# TODO: uncomment clasp when clasp 1.0.0 is packaged
# There's got to be a better way than this...
# The problem was that with --load everywhere, some
# implementations didn't exit with 0 on compilation failure
# Maybe a handler-case in buildScript?
sbcl = "${pkgs.sbcl}/bin/sbcl --script";
ecl = "${pkgs.ecl}/bin/ecl --shell";
abcl = ''${pkgs.abcl}/bin/abcl --batch --eval "(load \"$buildScript\")"'';
ccl = ''${pkgs.ccl}/bin/ccl --batch --eval "(load \"$buildScript\")" --'';
# clasp = ''${pkgs.clasp}/bin/clasp --non-interactive --quit --load'';
# Manually defined packages shadow the ones imported from quicklisp
sbclPackages = lispPackagesFor sbcl;
eclPackages = lispPackagesFor ecl;
abclPackages = lispPackagesFor abcl;
cclPackages = lispPackagesFor ccl;
# claspPackages = lispPackagesFor clasp;
sbclWithPackages = lispWithPackages sbcl;
eclWithPackages = lispWithPackages ecl;
abclWithPackages = lispWithPackages abcl;
cclWithPackages = lispWithPackages ccl;
# claspWithPackages = lispWithPackages clasp;
};
in commonLispPackages

View File

@ -0,0 +1,155 @@
# Assert that FILE exists and is executable
#
# assertExecutable FILE
assertExecutable() {
local file="$1"
[[ -f "$file" && -x "$file" ]] || \
die "Cannot wrap '$file' because it is not an executable file"
}
# construct an executable file that wraps the actual executable
# makeWrapper EXECUTABLE OUT_PATH ARGS
# ARGS:
# --argv0 NAME : set name of executed process to NAME
# (otherwise its called …-wrapped)
# --set VAR VAL : add VAR with value VAL to the executables
# environment
# --set-default VAR VAL : like --set, but only adds VAR if not already set in
# the environment
# --unset VAR : remove VAR from the environment
# --run COMMAND : run command before the executable
# --add-flags FLAGS : add FLAGS to invocation of executable
# --prefix ENV SEP VAL : suffix/prefix ENV with VAL, separated by SEP
# --suffix
# --prefix-each ENV SEP VALS : like --prefix, but VALS is a list
# --suffix-each ENV SEP VALS : like --suffix, but VALS is a list
# --prefix-contents ENV SEP FILES : like --suffix-each, but contents of FILES
# are read first and used as VALS
# --suffix-contents
makeWrapper() {
local original="$1"
local wrapper="$2"
local params varName value command separator n fileNames
local argv0 flagsBefore flags
assertExecutable "$original"
mkdir -p "$(dirname "$wrapper")"
echo "#! @shell@ -e" > "$wrapper"
params=("$@")
for ((n = 2; n < ${#params[*]}; n += 1)); do
p="${params[$n]}"
if [[ "$p" == "--set" ]]; then
varName="${params[$((n + 1))]}"
value="${params[$((n + 2))]}"
n=$((n + 2))
echo "export $varName=${value@Q}" >> "$wrapper"
elif [[ "$p" == "--set-default" ]]; then
varName="${params[$((n + 1))]}"
value="${params[$((n + 2))]}"
n=$((n + 2))
echo "export $varName=\${$varName-${value@Q}}" >> "$wrapper"
elif [[ "$p" == "--unset" ]]; then
varName="${params[$((n + 1))]}"
n=$((n + 1))
echo "unset $varName" >> "$wrapper"
elif [[ "$p" == "--run" ]]; then
command="${params[$((n + 1))]}"
n=$((n + 1))
echo "$command" >> "$wrapper"
elif [[ ("$p" == "--suffix") || ("$p" == "--prefix") ]]; then
varName="${params[$((n + 1))]}"
separator="${params[$((n + 2))]}"
value="${params[$((n + 3))]}"
n=$((n + 3))
if test -n "$value"; then
if test "$p" = "--suffix"; then
echo "export $varName=\$$varName\${$varName:+${separator@Q}}${value@Q}" >> "$wrapper"
else
echo "export $varName=${value@Q}\${$varName:+${separator@Q}}\$$varName" >> "$wrapper"
fi
fi
elif [[ "$p" == "--prefix-each" ]]; then
varName="${params[$((n + 1))]}"
separator="${params[$((n + 2))]}"
values="${params[$((n + 3))]}"
n=$((n + 3))
for value in $values; do
echo "export $varName=${value@Q}\${$varName:+${separator@Q}}\$$varName" >> "$wrapper"
done
elif [[ "$p" == "--suffix-each" ]]; then
varName="${params[$((n + 1))]}"
separator="${params[$((n + 2))]}"
values="${params[$((n + 3))]}"
n=$((n + 3))
for value in $values; do
echo "export $varName=\$$varName\${$varName:+$separator}${value@Q}" >> "$wrapper"
done
elif [[ ("$p" == "--suffix-contents") || ("$p" == "--prefix-contents") ]]; then
varName="${params[$((n + 1))]}"
separator="${params[$((n + 2))]}"
fileNames="${params[$((n + 3))]}"
n=$((n + 3))
for fileName in $fileNames; do
contents="$(cat "$fileName")"
if test "$p" = "--suffix-contents"; then
echo "export $varName=\$$varName\${$varName:+$separator}${contents@Q}" >> "$wrapper"
else
echo "export $varName=${contents@Q}\${$varName:+$separator}\$$varName" >> "$wrapper"
fi
done
elif [[ "$p" == "--add-flags" ]]; then
flags="${params[$((n + 1))]}"
n=$((n + 1))
flagsBefore="$flagsBefore $flags"
elif [[ "$p" == "--argv0" ]]; then
argv0="${params[$((n + 1))]}"
n=$((n + 1))
else
die "makeWrapper doesn't understand the arg $p"
fi
done
echo exec ${argv0:+-a \"$argv0\"} \""$original"\" \
"$flagsBefore" '"$@"' >> "$wrapper"
chmod +x "$wrapper"
}
addSuffix() {
suffix="$1"
shift
for name in "$@"; do
echo "$name$suffix"
done
}
filterExisting() {
for fn in "$@"; do
if test -e "$fn"; then
echo "$fn"
fi
done
}
# Syntax: wrapProgram <PROGRAM> <MAKE-WRAPPER FLAGS...>
wrapProgram() {
local prog="$1"
local hidden
assertExecutable "$prog"
hidden="$(dirname "$prog")/.$(basename "$prog")"-wrapped
while [ -e "$hidden" ]; do
hidden="${hidden}_"
done
mv "$prog" "$hidden"
# Silence warning about unexpanded $0:
# shellcheck disable=SC2016
makeWrapper "$hidden" "$prog" --argv0 '$0' "${@:2}"
}

View File

@ -0,0 +1,369 @@
{ build-asdf-system, lisp, quicklispPackagesFor, fixupFor, pkgs, ... }:
let
inherit (pkgs.lib)
head
makeLibraryPath
makeSearchPath
setAttr
hasAttr
optionals
hasSuffix
splitString
;
# Used by builds that would otherwise attempt to write into storeDir.
#
# Will run build two times, keeping all files created during the
# first run, exept the FASL's. Then using that directory tree as the
# source of the second run.
#
# E.g. cl-unicode creating .txt files during compilation
build-with-compile-into-pwd = args:
let
build = (build-asdf-system (args // { version = args.version + "-build"; }))
.overrideAttrs(o: {
buildPhase = with builtins; ''
mkdir __fasls
export LD_LIBRARY_PATH=${makeLibraryPath o.nativeLibs}:$LD_LIBRARY_PATH
export CLASSPATH=${makeSearchPath "share/java/*" o.javaLibs}:$CLASSPATH
export CL_SOURCE_REGISTRY=$CL_SOURCE_REGISTRY:$(pwd)//
export ASDF_OUTPUT_TRANSLATIONS="$(pwd):$(pwd)/__fasls:${storeDir}:${storeDir}"
${o.lisp} ${o.buildScript}
'';
installPhase = ''
mkdir -pv $out
rm -rf __fasls
cp -r * $out
'';
});
in build-asdf-system (args // {
# Patches are already applied in `build`
patches = [];
src = build;
});
# A little hacky
isJVM = hasSuffix "abcl" (head (splitString " " lisp));
# Makes it so packages imported from Quicklisp can be re-used as
# lispLibs ofpackages in this file.
ql = quicklispPackagesFor { inherit lisp; fixup = fixupFor packages; };
packages = rec {
asdf = build-with-compile-into-pwd {
pname = "asdf";
version = "3.3.5.3";
src = pkgs.fetchzip {
url = "https://gitlab.common-lisp.net/asdf/asdf/-/archive/3.3.5.3/asdf-3.3.5.3.tar.gz";
sha256 = "0aw200awhg58smmbdmz80bayzmbm1a6547gv0wmc8yv89gjqldbv";
};
systems = [ "asdf" "uiop" ];
};
uiop = asdf.overrideLispAttrs(o: {
pname = "uiop";
});
cffi = let
jna = pkgs.fetchMavenArtifact {
groupId = "net.java.dev.jna";
artifactId = "jna";
version = "5.9.0";
sha256 = "0qbis8acv04fi902qzak1mbagqaxcsv2zyp7b8y4shs5nj0cgz7a";
};
in build-asdf-system {
src = pkgs.fetchzip {
url = "http://beta.quicklisp.org/archive/cffi/2021-04-11/cffi_0.24.1.tgz";
sha256 = "17ryim4xilb1rzxydfr7595dnhqkk02lmrbkqrkvi9091shi4cj3";
};
version = "0.24.1";
pname = "cffi";
lispLibs = with ql; [
alexandria
babel
trivial-features
];
javaLibs = optionals isJVM [ jna ];
};
cffi-libffi = ql.cffi-libffi.overrideLispAttrs (o: {
src = pkgs.fetchzip {
url = "https://github.com/cffi/cffi/archive/3f842b92ef808900bf20dae92c2d74232c2f6d3a.tar.gz";
sha256 = "1jilvmbbfrmb23j07lwmkbffc6r35wnvas5s4zjc84i856ccclm2";
};
});
cl-unicode = build-with-compile-into-pwd {
pname = "cl-unicode";
version = "0.1.6";
src = pkgs.fetchzip {
url = "https://github.com/edicl/cl-unicode/archive/refs/tags/v0.1.6.tar.gz";
sha256 = "0ykx2s9lqfl74p1px0ik3l2izd1fc9jd1b4ra68s5x34rvjy0hza";
};
systems = [ "cl-unicode" ];
lispLibs = with ql; [
cl-ppcre
flexi-streams
];
};
quri = build-asdf-system {
src = pkgs.stdenv.mkDerivation {
pname = "patched";
version = "source";
src = pkgs.fetchzip {
url = "http://beta.quicklisp.org/archive/quri/2021-04-11/quri-20210411-git.tgz";
sha256 = "1pkvpiwwhx2fcknr7x47h7036ypkg8xzsskqbl5z315ipfmi8s2m";
};
# fix build with ABCL
buildPhase = ''
sed -i "s,[#][.](asdf.*,#P\"$out/data/effective_tld_names.dat\")," src/etld.lisp
'';
installPhase = ''
mkdir -pv $out
cp -r * $out
'';
};
version = "20210411";
pname = "quri";
lispLibs = with ql; [
alexandria
babel
cl-utilities
split-sequence
];
};
jzon = build-asdf-system {
src = pkgs.fetchzip {
url = "https://github.com/Zulu-Inuoe/jzon/archive/6b201d4208ac3f9721c461105b282c94139bed29.tar.gz";
sha256 = "01d4a78pjb1amx5amdb966qwwk9vblysm1li94n3g26mxy5zc2k3";
};
version = "0.0.0-20210905-6b201d4208";
pname = "jzon";
lispLibs = [
ql.closer-mop
];
systems = [ "com.inuoe.jzon" ];
};
cl-notify = build-asdf-system {
pname = "cl-notify";
version = "20080904-138ca7038";
src = pkgs.fetchzip {
url = "https://repo.or.cz/cl-notify.git/snapshot/138ca703861f4a1fbccbed557f92cf4d213668a1.tar.gz";
sha256 = "0k6ns6fzvjcbpsqgx85r4g5m25fvrdw9481i9vyabwym9q8bbqwx";
};
lispLibs = [
cffi
];
nativeLibs = [
pkgs.libnotify
];
};
tuple = build-asdf-system {
pname = "tuple";
version = "b74bd067d";
src = pkgs.fetchzip {
url = "https://fossil.galkowski.xyz/tuple/tarball/b74bd067d4533ac0/tuple.tar.gz";
sha256 = "0dk356vkv6kwwcmc3j08x7143549m94rd66rpkzq8zkb31cg2va8";
};
};
cl-tar-file = build-asdf-system {
pname = "cl-tar-file";
version = "v0.2.1";
src = pkgs.fetchzip {
url = let
rev = "0c10bc82f14702c97a26dc25ce075b5d3a2347d1";
in "https://gitlab.common-lisp.net/cl-tar/cl-tar-file/-/archive/${rev}/cl-tar-file-${rev}.tar.gz";
sha256 = "0i8j05fkgdqy4c4pqj0c68sh4s3klpx9kc5wp73qwzrl3xqd2svy";
};
lispLibs = with ql; [
alexandria
babel
trivial-gray-streams
_40ants-doc
salza2
chipz
flexi-streams
parachute
];
systems = [ "tar-file" "tar-file/test" ];
};
cl-tar = build-asdf-system {
pname = "cl-tar";
version = "v0.2.1";
src = pkgs.fetchzip {
url = let
rev = "7c6e07a10c93d9e311f087b5f6328cddd481669a";
in "https://gitlab.common-lisp.net/cl-tar/cl-tar/-/archive/${rev}/cl-tar-${rev}.tar.gz";
sha256 = "0wp23cs3i6a89dibifiz6559la5nk58d1n17xvbxq4nrl8cqsllf";
};
lispLibs = with ql; [
alexandria
babel
local-time
split-sequence
_40ants-doc
parachute
osicat
] ++ [ cl-tar-file ];
systems = [
"tar"
"tar/common-extract"
"tar/simple-extract"
"tar/extract"
"tar/create"
"tar/docs"
"tar/test"
"tar/create-test"
"tar/extract-test"
"tar/simple-extract-test"
];
};
cl-fuse = build-with-compile-into-pwd {
inherit (ql.cl-fuse) pname version src lispLibs;
nativeBuildInputs = [ pkgs.fuse ];
nativeLibs = [ pkgs.fuse ];
};
cl-containers = build-asdf-system {
inherit (ql.cl-containers) pname version src;
lispLibs = ql.cl-containers.lispLibs ++ [ ql.moptilities ];
systems = [ "cl-containers" "cl-containers/with-moptilities" ];
};
swank = build-with-compile-into-pwd {
inherit (ql.swank) pname version src lispLibs;
patches = [ ./patches/swank-pure-paths.patch ];
postConfigure = ''
substituteAllInPlace swank-loader.lisp
'';
};
clx-truetype = build-asdf-system {
pname = "clx-truetype";
version = "20160825-git";
src = pkgs.fetchzip {
url = "http://beta.quicklisp.org/archive/clx-truetype/2016-08-25/clx-truetype-20160825-git.tgz";
sha256 = "079hyp92cjkdfn6bhkxsrwnibiqbz4y4af6nl31lzw6nm91j5j37";
};
lispLibs = with ql; [
alexandria bordeaux-threads cl-aa cl-fad cl-paths cl-paths-ttf
cl-store cl-vectors clx trivial-features zpb-ttf
];
};
mgl = build-asdf-system {
pname = "mgl";
version = "2021-10-07";
src = pkgs.fetchzip {
url = "https://github.com/melisgl/mgl/archive/e697791a9bcad3b6e7b3845246a2aa55238cfef7.tar.gz";
sha256 = "09sf7nq7nmf9q7bh3a5ygl2i2n0nhrx5fk2kv5ili0ckv7g9x72s";
};
lispLibs = with ql; [
alexandria closer-mop array-operations lla cl-reexport mgl-pax
named-readtables pythonic-string-reader
] ++ [ mgl-mat ];
systems = [ "mgl" "mgl/test" ];
};
mgl-mat = build-asdf-system {
pname = "mgl-mat";
version = "2021-10-11";
src = pkgs.fetchzip {
url = "https://github.com/melisgl/mgl-mat/archive/3710858bc876b1b86e50f1db2abe719e92d810e7.tar.gz";
sha256 = "1aa2382mi55rp8pd31dz4d94yhfzh30vkggcvmvdfrr4ngffj0dx";
};
lispLibs = with ql; [
alexandria bordeaux-threads cffi cffi-grovel cl-cuda
flexi-streams ieee-floats lla mgl-pax static-vectors
trivial-garbage cl-fad
];
systems = [ "mgl-mat" "mgl-mat/test" ];
};
mathkit = build-asdf-system {
inherit (ql.mathkit) pname version src asds lisp;
lispLibs = ql.mathkit.lispLibs ++ [ ql.sb-cga ];
};
nyxt-gtk = build-asdf-system {
inherit (ql.nyxt) pname lisp;
version = "2.2.4";
lispLibs = ql.nyxt.lispLibs ++ (with ql; [
cl-cffi-gtk cl-webkit2 mk-string-metrics
]);
src = pkgs.fetchzip {
url = "https://github.com/atlas-engineer/nyxt/archive/2.2.4.tar.gz";
sha256 = "12l7ir3q29v06jx0zng5cvlbmap7p709ka3ik6x29lw334qshm9b";
};
buildInputs = [
pkgs.makeWrapper
# needed for GSETTINGS_SCHEMAS_PATH
pkgs.gsettings-desktop-schemas pkgs.glib pkgs.gtk3
# needed for XDG_ICON_DIRS
pkgs.gnome.adwaita-icon-theme
];
buildScript = pkgs.writeText "build-nyxt.lisp" ''
(require :asdf)
(asdf:load-system :nyxt/gtk-application)
(sb-ext:save-lisp-and-die "nyxt" :executable t
#+sb-core-compression :compression
#+sb-core-compression t
:toplevel #'nyxt:entry-point)
'';
# Run with WEBKIT_FORCE_SANDBOX=0 if getting a runtime error in webkitgtk-2.34.4
installPhase = ql.nyxt.installPhase + ''
rm -v $out/nyxt
mkdir -p $out/bin
cp -v nyxt $out/bin
wrapProgram $out/bin/nyxt \
--prefix LD_LIBRARY_PATH : $LD_LIBRARY_PATH \
--prefix XDG_DATA_DIRS : $XDG_ICON_DIRS \
--prefix XDG_DATA_DIRS : $GSETTINGS_SCHEMAS_PATH \
--prefix GIO_EXTRA_MODULES ":" ${pkgs.dconf.lib}/lib/gio/modules/ \
--prefix GIO_EXTRA_MODULES ":" ${pkgs.glib-networking}/lib/gio/modules/
'';
};
nyxt = nyxt-gtk;
ltk = ql.ltk.overrideLispAttrs (o: {
src = pkgs.fetchzip {
url = "https://github.com/uthar/ltk/archive/f19162e76d6c7c2f51bd289b811d9ba20dd6555e.tar.gz";
sha256 = "0mzikv4abq9yqlj6dsji1wh34mjizr5prv6mvzzj29z1485fh1bj";
};
version = "f19162e76";
});
s-sql_slash_tests = ql.s-sql_slash_tests.overrideLispAttrs (o: {
lispLibs = o.lispLibs ++ [
ql.cl-postgres_slash_tests
];
});
simple-date_slash_postgres-glue = ql.simple-date_slash_postgres-glue.overrideLispAttrs (o: {
lispLibs = o.lispLibs ++ [
ql.cl-postgres_slash_tests
];
});
};
in packages

View File

@ -0,0 +1,28 @@
Prevent swank from attempting write into storeDir
--- a/swank-loader.lisp
+++ b/swank-loader.lisp
@@ -162,7 +162,7 @@
,(unique-dir-name)))
(user-homedir-pathname)))
-(defvar *fasl-directory* (default-fasl-dir)
+(defvar *fasl-directory* #P"@out@/fasl/"
"The directory where fasl files should be placed.")
(defun binary-pathname (src-pathname binary-dir)
@@ -284,12 +284,7 @@
(contrib-dir src-dir))))
(defun delete-stale-contrib-fasl-files (swank-files contrib-files fasl-dir)
- (let ((newest (reduce #'max (mapcar #'file-write-date swank-files))))
- (dolist (src contrib-files)
- (let ((fasl (binary-pathname src fasl-dir)))
- (when (and (probe-file fasl)
- (<= (file-write-date fasl) newest))
- (delete-file fasl))))))
+ (declare (ignore swank-files contrib-files fasl-dir)))
(defun compile-contribs (&key (src-dir (contrib-dir *source-directory*))
(fasl-dir (contrib-dir *fasl-directory*))
Diff finished. Sat Jan 22 23:57:27 2022

View File

@ -0,0 +1,6 @@
(require :asdf)
(pushnew (truename "./import") asdf:*central-registry*)
(asdf:load-system :org.lispbuilds.nix)
(load "./import/main.lisp")
(org.lispbuilds.nix/main::main)

View File

@ -0,0 +1,140 @@
{ pkgs, build-asdf-system, fixup ? pkgs.lib.id, ... }:
with pkgs;
with lib;
with lib.lists;
with lib.strings;
let
# FIXME: automatically add nativeLibs based on conditions signalled
extras = {
"cl+ssl" = pkg: {
nativeLibs = [ openssl ];
};
cl-cffi-gtk-glib = pkg: {
nativeLibs = [ glib ];
};
cl-cffi-gtk-cairo = pkg: {
nativeLibs = [ cairo ];
};
cl-cffi-gtk-gdk = pkg: {
nativeLibs = [ gtk3 ];
};
cl-cffi-gtk-gdk-pixbuf = pkg: {
nativeLibs = [ gdk-pixbuf ];
};
cl-cffi-gtk-pango = pkg: {
nativeLibs = [ pango ];
};
cl-gobject-introspection = pkg: {
nativeLibs = [ glib gobject-introspection ];
};
cl-mysql = pkg: {
nativeLibs = [ mysql-client ];
};
clsql-postgresql = pkg: {
nativeLibs = [ postgresql.lib ];
};
clsql-sqlite3 = pkg: {
nativeLibs = [ sqlite ];
};
cl-webkit2 = pkg: {
nativeLibs = [ webkitgtk ];
};
dbd-mysql = pkg: {
nativeLibs = [ mysql-client ];
};
lla = pkg: {
nativeLibs = [ openblas ];
};
cffi-libffi = pkg: {
nativeBuildInputs = [ libffi ];
nativeLibs = [ libffi ];
};
cl-rabbit = pkg: {
nativeBuildInputs = [ rabbitmq-c ];
nativeLibs = [ rabbitmq-c ];
};
trivial-ssh-libssh2 = pkg: {
nativeLibs = [ libssh2 ];
};
sqlite = pkg: {
nativeLibs = [ sqlite ];
};
cl-libuv = pkg: {
nativeBuildInputs = [ libuv ];
nativeLibs = [ libuv ];
};
cl-liballegro = pkg: {
# build doesnt fail without this, but fails on runtime
# weird...
nativeLibs = [ allegro5 ];
};
classimp = pkg: {
nativeLibs = [ assimp ];
};
sdl2 = pkg: {
nativeLibs = [ SDL2 ];
};
lispbuilder-sdl-cffi = pkg: {
nativeLibs = [ SDL ];
};
cl-opengl = pkg: {
nativeLibs = [ libGL ];
};
cl-glu = pkg: {
nativeLibs = [ libGLU ];
};
cl-glut = pkg: {
nativeLibs = [ freeglut ];
};
cl-glfw = pkg: {
nativeLibs = [ glfw ];
};
cl-glfw-opengl-core = pkg: {
nativeLibs = [ libGL ];
};
cl-glfw3 = pkg: {
nativeLibs = [ glfw ];
};
lev = pkg: {
nativeLibs = [ libev ];
};
cl-rdkafka = pkg: {
nativeBuildInputs = [ rdkafka ];
nativeLibs = [ rdkafka ];
};
cl-async-ssl = pkg: {
nativeLibs = [ openssl ];
};
osicat = pkg: {
LD_LIBRARY_PATH = "${pkg}/posix/";
};
iolib = pkg: {
nativeBuildInputs = [ libfixposix ];
nativeLibs = [ libfixposix ];
systems = [ "iolib" "iolib/os" "iolib/pathnames" ];
};
};
qlpkgs =
if builtins.pathExists ./imported.nix
# then filterAttrs (n: v: all (check: !(check n v)) broken) (import ./imported.nix { inherit pkgs; })
then import ./imported.nix { inherit (pkgs) runCommand fetchzip; pkgs = builtQlpkgs; }
else {};
builtQlpkgs = mapAttrs (n: v: build v) qlpkgs;
build = pkg:
let
builtPkg = build-asdf-system pkg;
withExtras = pkg //
(optionalAttrs
(hasAttr pkg.pname extras)
(extras.${pkg.pname} builtPkg));
fixedUp = fixup withExtras;
in build-asdf-system fixedUp;
in builtQlpkgs

View File

@ -0,0 +1,34 @@
# lisp-modules
Utilities for packaging ASDF systems using Nix.
## Quick start
#### Build an ASDF system:
```
nix-build ./examples/bordeaux-threads.nix
ls result/src
```
#### Build an `sbclWithPackages`:
```
nix-build ./examples/sbcl-with-bt.nix
result/bin/sbcl
```
#### Re-import Quicklisp packages:
```
nix-shell --run 'sbcl --script ql-import.lisp'
```
#### Test build of packages
```
(cd test; sbcl --script test.lisp ccl)
```
## Documentation
See `doc` directory.

View File

@ -0,0 +1,7 @@
with import ../../../default.nix {};
mkShell {
nativeBuildInputs = [
(lispPackages_new.sbclWithPackages
(ps: with ps; [ alexandria str dexador cl-ppcre sqlite arrow-macros jzon ]))
];
}

View File

@ -0,0 +1,395 @@
_1am
_3bmd
_3bmd-ext-code-blocks
access
acclimation
agutil
alexandria
anaphora
arnesi
array-operations
array-utils
arrows
asdf
asdf-package-system
asdf-system-connections
babel
binomial-heap
binpack
blackbird
bordeaux-threads
buildnode
buildnode-xhtml
calispel
cffi
cffi-grovel
cffi-toolchain
cffi-uffi-compat
chanl
check-it
chipz
chunga
circular-streams
cl-aa
cl-annot
cl-anonfun
cl-ansi-text
cl-async
cl-async-base
cl-async-repl
cl-async-ssl
cl-async-util
cl-base64
cl-cffi-gtk
cl-cffi-gtk-cairo
cl-cffi-gtk-gdk
cl-cffi-gtk-gdk-pixbuf
cl-cffi-gtk-gio
cl-cffi-gtk-glib
cl-cffi-gtk-gobject
cl-cffi-gtk-pango
cl-change-case
cl-cli
cl-colors
cl-colors2
cl-containers
cl-cookie
cl-css
cl-csv
cl-cuda
cl-custom-hash-table
cl-dbi
cl-difflib
cl-digraph
cl-dot
cl-emb
cl-environments
cl-fad
cl-form-types
cl-fuse
cl-fuse-meta-fs
cl-fuzz
cl-geometry
cl-gobject-introspection
cl-heap
cl-hooks
cl-html-diff
cl-html-parse
cl-html5-parser
cl-interpol
cl-jpeg
cl-json
cl-l10n
cl-l10n-cldr
cl-libuv
cl-locale
cl-markup
cl-mustache
cl-mysql
cl-num-utils
cl-paths
cl-paths-ttf
cl-pattern
cl-pdf
cl-postgres
cl-postgres_plus_local-time
cl-postgres_slash_tests
cl-ppcre
cl-ppcre-template
cl-ppcre-test
cl-ppcre-unicode
cl-prevalence
cl-protobufs
cl-qprint
cl-qrencode
cl-reexport
cl-shellwords
cl-slice
cl-smt-lib
cl-smtp
cl-speedy-queue
cl-store
cl-svg
cl-syntax
cl-syntax-annot
cl-syntax-anonfun
cl-syntax-markup
cl-syslog
cl-test-more
cl-typesetting
cl-unicode
cl-unification
cl-utilities
cl-vectors
cl-webkit2
cl-who
cl-xmlspam
cl_plus_ssl
clack
clack-socket
classowary
clfswm
closer-mop
closure-common
closure-html
clsql
clsql-postgresql
clsql-postgresql-socket
clsql-sqlite3
clsql-uffi
clss
cluffer
clump
clump-2-3-tree
clump-binary-tree
clunit
clunit2
clx
clx-truetype
collectors
colorize
command-line-arguments
css-lite
css-selectors
css-selectors-simple-tree
css-selectors-stp
cxml
cxml-stp
cxml_slash_test
data-table
dbd-mysql
dbd-postgres
dbd-sqlite3
dbi
dbi-test
dbus
defclass-std
dexador
dissect
djula
do-urlencode
documentation-utils
drakma
eager-future2
enchant
esrap
esrap-peg
external-program
fare-csv
fare-mop
fare-quasiquote
fare-quasiquote-extras
fare-quasiquote-optima
fare-quasiquote-readtable
fare-utils
fast-http
fast-io
fiasco
file-attributes
fiveam
flexi-streams
float-features
flow
fn
form-fiddle
fset
generic-cl
generic-cl_dot_arithmetic
generic-cl_dot_collector
generic-cl_dot_comparison
generic-cl_dot_container
generic-cl_dot_internal
generic-cl_dot_iterator
generic-cl_dot_lazy-seq
generic-cl_dot_map
generic-cl_dot_math
generic-cl_dot_object
generic-cl_dot_sequence
generic-cl_dot_set
gettext
global-vars
glsl-docs
glsl-spec
glsl-symbols
heap
html-encode
http-body
hu_dot_dwim_dot_asdf
hu_dot_dwim_dot_common
hu_dot_dwim_dot_common-lisp
hu_dot_dwim_dot_def
hu_dot_dwim_dot_def_plus_swank
hu_dot_dwim_dot_defclass-star
hu_dot_dwim_dot_stefil
hu_dot_dwim_dot_stefil_plus_hu_dot_dwim_dot_def
hu_dot_dwim_dot_stefil_plus_hu_dot_dwim_dot_def_plus_swank
hu_dot_dwim_dot_stefil_plus_swank
hunchensocket
hunchentoot
idna
ieee-floats
inferior-shell
introspect-environment
iolib
iolib_dot_asdf
iolib_dot_base
iolib_dot_common-lisp
iolib_dot_conf
ironclad
iterate
jonathan
jpl-queues
jpl-util
jsown
kmrcl
lack
lack-component
lack-middleware-backtrace
lack-util
lambda-fiddle
legit
let-plus
lev
lfarm-client
lfarm-common
lfarm-server
lfarm-ssl
lift
lisp-binary
lisp-namespace
lisp-unit
lisp-unit2
lla
local-time
log4cl
lparallel
lquery
marshal
md5
metabang-bind
metatilities-base
mgl
mgl-mat
mgl-pax
minheap
misc-extensions
mk-string-metrics
mmap
moptilities
more-conditions
mt19937
named-readtables
nbd
net-telent-date
net_dot_didierverna_dot_asdf-flv
nibbles
nyxt
optima
osicat
parachute
parenscript
parse-declarations-1_dot_0
parse-float
parse-number
parseq
parser-combinators
parser_dot_common-rules
pcall
pcall-queue
physical-quantities
plump
postmodern
proc-parse
prove
prove-asdf
ptester
puri
pythonic-string-reader
quasiquote-2_dot_0
query-fs
quri
rfc2388
rove
rt
s-sql
s-sql_slash_tests
s-sysdeps
s-xml
salza2
serapeum
simple-date
simple-date-time
simple-date_slash_postgres-glue
simple-inferiors
simple-tasks
slynk
smart-buffer
smug
spinneret
split-sequence
sqlite
static-dispatch
static-vectors
stefil
str
string-case
stumpwm
swank
swap-bytes
sycamore
symbol-munger
trees
trivia
trivia_dot_balland2006
trivia_dot_level0
trivia_dot_level1
trivia_dot_level2
trivia_dot_quasiquote
trivia_dot_trivial
trivial-arguments
trivial-backtrace
trivial-clipboard
trivial-cltl2
trivial-features
trivial-file-size
trivial-garbage
trivial-gray-streams
trivial-indent
trivial-macroexpand-all
trivial-main-thread
trivial-mimes
trivial-package-local-nicknames
trivial-shell
trivial-types
trivial-utf-8
trivial-with-current-source-form
type-i
uax-15
uffi
uiop
unit-test
unix-options
unix-opts
usocket
usocket-server
utilities_dot_print-items
utilities_dot_print-tree
uuid
varjo
vas-string-metrics
vecto
vom
wild-package-inferred-system
woo
wookie
xembed
xkeyboard
xml_dot_location
xmls
xpath
xpath_slash_test
xsubseq
yacc
yason
zpb-ttf
zpng

View File

@ -0,0 +1,94 @@
#!/usr/bin/env -S sbcl --script
(require :uiop)
;; prevent glibc hell
(setf (uiop:getenv "LD_LIBRARY_PATH") "")
(defparameter packages (uiop:read-file-lines "./lispPackagesToTest.txt"))
(defparameter lisp (or (cadr sb-ext:*posix-argv*) "sbcl"))
(defparameter nix-build "nix-build -E 'with import ../../../../default.nix {}; lispPackages_new.~aPackages.~a'")
(defparameter cpu-count
(length
(remove-if-not
(lambda (line)
(uiop:string-prefix-p "processor" line))
(uiop:read-file-lines "/proc/cpuinfo"))))
(defparameter sem (sb-thread:make-semaphore :count cpu-count))
(defparameter statuses (make-hash-table :synchronized t))
(defparameter log-lock (sb-thread:make-mutex))
(format *error-output* "Testing ~a on ~a cores~%" lisp cpu-count)
(defun clear-line ()
(write-char #\Return *error-output*)
(write-char #\Escape *error-output*)
(write-char #\[ *error-output*)
(write-char #\K *error-output*))
(declaim (type fixnum errors))
(defglobal errors 0)
(defmacro when-let (bindings &rest body)
(reduce
(lambda (expansion form)
(destructuring-bind (var test) form
(let ((testsym (gensym (symbol-name var))))
`(let ((,testsym ,test))
(when ,testsym
(let ((,var ,testsym))
,expansion))))))
(reverse bindings)
:initial-value `(progn ,@body)))
(dolist (pkg packages)
(sb-thread:wait-on-semaphore sem)
(sb-thread:make-thread
(lambda ()
(handler-case
(unwind-protect
(multiple-value-bind (out err code)
(uiop:run-program
(format nil nix-build lisp pkg)
:error-output '(:string :stripped t)
:ignore-error-status t)
(declare (ignorable err))
(setf (gethash pkg statuses) code)
(when-let ((pos (search "LOAD-FOREIGN-LIBRARY-ERROR" err :test #'string=))
(lines (uiop:split-string (subseq err pos) :separator '(#\Newline))))
(setf (gethash pkg statuses)
(fourth lines)))
(sb-thread:with-mutex (log-lock)
(clear-line)
(format *error-output* "[~a/~a] ~[OK~:;ERROR~] ~a~[~:;~%~]"
(hash-table-count statuses)
(length packages)
code
pkg
code)
(force-output *error-output*))
(unless (zerop code)
(sb-ext:atomic-incf errors)))
(sb-thread:signal-semaphore sem))
(error (e)
(format t "~a~%" e)
(sb-ext:quit :recklessly-p t :unix-status 1))))))
(sb-thread:wait-on-semaphore sem :n cpu-count)
(format t "~%Done (~a/~a)."
(- (length packages) errors)
(length packages))
(when (plusp errors)
(format t "~%~%~a Errors: " errors)
(maphash (lambda (k v)
(unless (and (numberp v) (zerop v))
(format t "~% ~a: ~a" k v)))
statuses))

View File

@ -21331,6 +21331,9 @@ with pkgs;
quicklispPackagesGCL = dontRecurseIntoAttrs (quicklispPackagesFor (wrapLisp gcl));
quicklispPackages = quicklispPackagesSBCL;
# Alternative lisp-modules implementation
lispPackages_new = callPackage ../development/lisp-modules-new/lisp-packages.nix {};
### DEVELOPMENT / PERL MODULES
perlInterpreters = callPackages ../development/interpreters/perl {};