Clementson's Blog: Managing CL Libraries

Managing CL Libraries

Tuesday, October 4, 2005

Like a lot of CL developers, I do all my Lisp development work in Emacs with SLIME. Since I work with multiple CL implementations (although I usually use OpenMCL since it's a really convenient implementation to use on Mac OS X), I maintain a .swank.lisp SLIME initialization file that caters for the different implementations so that I have a similar set of lisp libraries available regardless of which implementation I am using. The two key packages that I use to manage CL libraries are

asdf: The use of asdf is pretty much a no-brainer since most "modern" packages have an asdf system definition that can be used to build/load the library.
asdf-install: I use asdf-install since it makes downloading a package and all it's prerequisites very simple. Edi Weitz provides an excellent tutorial for using asdf-install; however, it is really simple to setup and use.

So, the combination of the two (asdf-install to get and install the package and asdf to manage the build/load of the package) works very nicely. However, the use of asdf-install depends on there being a tarball that can be downloaded and installed. Although most packages that I use fall into this category, there are also a number of packages for which I'm always using the "bleeding-edge" versions from CVS (or, lately, Subversion, arch or darcs). I dislike having to do manual setup; therefore, I have a few configuration things that I do to manage the combination of "stable" and "unstable" libraries. First of all, I setup my lisp directory structure as follows:

~/: My home directory (which is /Users/bc/ on my Mac).

bin/: I put the install directories for SBCL, OpenMCL, ACL, LispWorks, CLISP in ~/bin/ rather than in /usr/bin/ as many other people do as I prefer to have them visible to my user profile only.
lisp/: The ~/lisp/ directory is the "root" directory for all lisp libraries and source code.
- docs/: The location of my local copy of the CL Hyperspec and a number of other lisp-related documents.
- site/: Contains all of the directories of lisp libraries that are installed via asdf-install.
- src/: Contains directories with lisp source code that I keep for reference only or that can't be loaded with asdf.
- systems/: Contains the symbolic links to the .asd files that are in the ~/lisp/site/* directories. These symbolic links are created automatically by asdf-install.
- unstable/: Contains directories of "bleeding-edge" versions of lisp libraries that I have downloaded from version control repositories.

So, I always have symbolic links (created automatically using asdf-install) pointing to the current versions of "stable" CL packages that I'm using. I dynamically create pointers to the "unstable" libraries in my .swank.lisp initialization file (using a technique that I discussed before in my blog). That way, I can swap and change things in my "unstable" directories and not mess around with creating symbolic links for things that will probably not be around for too long (or which might have their names/locations changed). Then, it's simply a matter of doing an asdf load of any package!

For the curious, here are the relevant portions from my .swank.lisp initialization file:

(in-package :cl-user)


;; Root location for CL libraries
(defvar *lisp-dirs* "/Users/bc/lisp/" "Root location of CL library installs")


;; Some implemenations come with asdf, some don't
#-(or sbcl openmcl acl) (load (concatenate 'string *lisp-dirs* "src/asdf/asdf"))
#+(or sbcl openmcl acl) (require 'asdf)


;; Point to "stable" CL libraries
(push (concatenate 'string *lisp-dirs* "systems/") asdf:*central-registry*)


;; Point to "unstable" CL libraries
#+allegro (dolist (dir-candidate (directory (concatenate 'string *lisp-dirs* "unstable/*")))
	    (when (file-directory-p dir-candidate)
	      (let ((asd-candidate (merge-pathnames "*.asd" (pathname-as-directory dir-candidate))))
		(when (directory asd-candidate)
		  (push (pathname-as-directory dir-candidate) asdf:*central-registry*)))))


#+lispworks (dolist (dir-candidate (directory (concatenate 'string *lisp-dirs* "unstable/*")))
	      (when (lw:file-directory-p dir-candidate)
		(let ((asd-candidate (merge-pathnames "*.asd" dir-candidate)))
		  (when (directory asd-candidate)
		    (push dir-candidate asdf:*central-registry*)))))


#+clisp (dolist (dir-candidate (directory (concatenate 'string *lisp-dirs* "unstable/*/")))
	  (let ((asd-candidate (merge-pathnames "*.asd" dir-candidate)))
	    (when (directory asd-candidate)
	      (push dir-candidate asdf:*central-registry*))))


#+sbcl (dolist (dir-candidate (directory (concatenate 'string *lisp-dirs* "unstable/*/")))
	 (let ((asd-candidate (merge-pathnames "*.asd" dir-candidate)))
	   (when (directory asd-candidate)
	     (push dir-candidate asdf:*central-registry*))))


#+openmcl (dolist (dir-candidate (directory (concatenate 'string *lisp-dirs* "unstable/*") :directories t))
	    (let ((asd-candidate (merge-pathnames "*.asd" dir-candidate)))
	      (when (directory asd-candidate)
		(push dir-candidate asdf:*central-registry*))))


;; Common functions
(defun asdf (lib)
  (asdf:oos 'asdf:load-op lib))


(asdf :asdf-install)


;;; If the fasl was stale, try to recompile and load (once). 
#+(or sbcl allegro lispworks)
(defmethod asdf:perform :around ((o asdf:load-op) (c asdf:cl-source-file))
  (handler-case (call-next-method o c)
    (#+sbcl sb-ext:invalid-fasl 
      #+allegro excl::file-incompatible-fasl-error
      #+lispworks conditions:fasl-error ()
      #-(or sbcl allegro lispworks) error ()
      (asdf:perform (make-instance 'asdf:compile-op) c)
      (call-next-method))))

The last around-method re-compiles "stale" files when necessary. This is useful when you upgrade your CL implementation and need to rebuild compiled objects (sure, I could delete all fasl's and just do a load, but remember what I said about not liking to do any unnecessary setup stuff ;-) ). Incidentally, I discussed this "stale fasl" technique once before and Edi Weitz sent me an email with an explanation of how to do it in LispWorks. Does anyone know the equivalent incantations for CLISP and OpenMCL?

October 2005
Sun	Mon	Tue	Wed	Thu	Fri	Sat
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31
Sep Nov