Structuring large Lisp applications

Question

I am currently trying to wrap my head around packages, systems & co.

I now have read Packages, systems, modules, libraries - WTF? a few times, and I think I'm still having difficulties to get it right.

If I simply want to split a Lisp source file into two files, where one shall "use" the other - how do I do that? Do I need to build a system for this? Should I use a module? …? I'm coming from a Node.js background, and there you can simply say

var foo = require('./foo');

to get a reference to what's exported in file foo.js. What is the closest equivalent to this in Lisp?

I understand that ASDF is for systems, and that it is bundled as part of Quicklisp, at least according to its documentation:

ASDF comes bundled with all recent releases of active Common Lisp implementations as well as with quicklisp […]

Okay, Quicklisp is for libraries, but what is their relationship? Is Quicklisp something such as a "package manager" in other languages? And if so, then what exactly does ASDF provide?

Sorry for these many questions, but I think it just shows the trouble I have to understand how to structure Lisp applications. Any help would be greatly appreciated :-)

Answer 1

System

For structuring large system use a system management tool. A 'free' one is ASDF.

You would need a system declaration, which lists the parts of you library or application. Typically it goes into an own file. Then you load a system or compile a system. There should be tutorials how to do that.

A simple Lisp system might have the following files:

• a system file describing the system, its parts and whatever other stuff is needed (other systems)
• a package file which describes the namespaces used
• a basic tools file (for examples functions used by the macro)
• a macro file which lists the macros (used so that they get compiled/loaded before the rest of the software)
• one or more other files with functionality.

Quicklisp is independent of that. It's a software distribution tool.

Quick hack to compile and load files

But you can also compile and load files the old fashioned way without a system tool:

(defparameter *files*
  '("/yourdir/foo.lisp" "/yourdir/bar.lisp"))

(defun compile-foobar ()
  (mapc #'compile-file *files*))

(defun load-foobar ()
  (mapc #'load *files*))

(defun compile-and-load ()
  (mapc (lambda (file)
           (load (compile-file file)))
        *files*))

In reality there might be more to it, but often it's enough. It should be easy to write your own building tool. A typical system tool will provide many more features for building more complex software in a structured way. Many of the ideas for these tools reach back at least 35 years. See for example the Lisp Machine manual, here the edition from 1984, chapter Maintaining Large Systems.

The role of files

Note that in plain Common Lisp the role of files and its semantics are not very complex.

A file is not a namespace, it is not associated with a class/subclass or an object, it is not a module. You mix Lisp constructs in a file like you want. Files can be arbitrary large (for example one complex library has a version where it is delivered as single source file with 30000 lines). The only real place in the standard semantics where a file plays a role is when compiling a file. What side effects has compiling a file? What optimizations can a compiler do?

Other than that it is assumed that the development environment provides services like load and compiling groups of files aka systems, provide overviews of compilation errors, record source locations of definitions, can locate definitions and more. A tool like ASDF handles the system part.

Answer 2

There is a require function in Common Lisp, but it deprecated. If you simply want to split your code in one or more pieces, to use it interactively in the REPL, you can put the code in different files and then load each of them. If instead you want to write a full lisp project, I have found very useful the quickproject package, that provides a simple starting point for the creation of new packages.