Bill Clementson's Blog: clbuild - (Lisp with Batteries Included

clbuild - (Lisp with Batteries Included - Part 2)

Wednesday, January 16, 2008

Downloading (and keeping up-to-date) dependent packages in CL has always been a bit of a drag (not difficult to do, but a manual, boring process). A while back, I posted about the STARTER-PACK utility Edi Weitz had written. It provides an easy mechanism for quickly downloading/installing a lot of the more commonly used CL libraries (along with all their dependencies). I titled the post Lisp with Batteries Included. However, there were characteristics of Edi's utility that meant that it was not really suitable for everyone:

It was only "officially" supported on Windows (Edi accepted patches from me that allowed it to also work on Mac OS X)
It only ran on LispWorks and not other CL implementations
It only downloaded the latest released version of each library. Although this makes sense for a "starter pack" that is designed to get beginners up and running with commonly-used CL libraries, many experienced CL developers prefer to work with the "bleeding edge" versions of packages (e.g. - the source that is in the project's version control system, not the latest "official" release).

Another alternative has been ASDF-Install. ASDF-Install is now bundled by some CL implementations and it works with all major CL implementations. It's nice in that it will automatically download any dependencies and it supports almost all popular CL packages. However, it also has a number of limitations:

As with Edi's STARTER-PACK, ASDF-Install only downloaded the latest released version of each library.
ASDF-Install looks on CLiki to find where to download a package from. This is potentially insecure as CLiki has been "cracked" in the past and a link to an ASDF-Installable package could be modified to point to a different tarball which (when installed) might do bad things. This is a recognized limitation of ASDF-Install and there is support for GPG in ASDF-Install to help ensure that the tarball that is downloaded is the "real" one. However, many people find setting up and using GPG for use with ASDF-Install a hassle and just bypass the GPG checks. This makes ASDF-Install a really big "potential" security risk.

So, if you're a CL developer who wants to use "bleeding edge" packages, what are you to do? A recent post from Andreas Fuchs reminded me about clbuild (another CL package installer), so I had a play with it the other day. But, first of all, a bit of background: Luke Gorrie (one of the main developers of SLIME and a bunch of other neat stuff) wrote a shell script which he called "clbuild" (see here, here) to keep up-to-date with the bleeding edge versions of a lot of related packages. When Luke decided to go volunteer for OLPC in Nepal, clbuild was adopted and extended by some other CL hackers. It now has a permanent home on common-lisp.net. As befitting a "bleeding edge" package manager, the docs aren't quite completed yet and you need to use darcs to install it; however, it does a pretty good job of installing and keeping you up-to-date with quite a number of CL packages. Here's a list of what's currently supported (it's really easy to add additional packages to the list):

acl-compat		cl-pdf			closure			flexichain		parse-number		spatial-trees
alexandria		cl-ppcre		closure-common		graphic-forms		plexippus-xpath		split-sequence
babel			cl-typesetting		closure-html		gsharp			psgraph			trivial-features
beirc			cl-vectors		clsql			hunchentoot		puri			trivial-gray-streams
bordeaux-threads	cl-webdav		clx			iterate			rfc2388			trivial-sockets
cffi			cl-who			cxml			lw-compat		rt			url-rewrite
chunga			cl-yacc			cxml-rng		mcclim			salza			usocket
cl+ssl			clim-alerts		cxml-stp		md5			salza2			vecto
cl-base64		climacs			drakma			mel-base		sbcl			zip
cl-fad			climplayer		eclipse			midi			skippy			zpb-ttf
cl-irc			closer-mop		flexi-streams		net-telent-date		slime			zpng

You can use clbuild to download the latest version (either cvs, svn, darcs, git, or tarball) of any supported package (with the dependencies). It will also download and build the latest version of SBCL, serve as a "launcher" for a number of packages, and do a bunch of other things (see below). Although SBCL is the primary CL implementation catered for by clbuild, OpenMCL/CCL and CLISP (and perhaps other CL implementations?) can be used as well (SBCL/Linux is the configuration that clbuild is tested with the most; however, I didn't have any problems with clbuild using SBCL on Mac OS X.). Here are the available options (an edited version of "clbuild help"):

Usage:

  check         check availability of all necessary helper applications

  update [--dependencies|--no-dependencies] PROJECT_NAME
                download/update this project
  update [--resume]
                download/update all projects.  With --resume, consider
                only projects that a previous update run failed to fetch.
  skip PROJECT_NAME
                mark this project as done for the purposes of update --resume

  recompile [PROJECT_NAME] 
                recompile all applications (or just PROJECT_NAME)
  dumpcore [PROJECT_NAME]  
                recompile and dump a core file for faster startup
  build [PROJECT_NAME]     
                update && dumpcore

  diff          show local changes (for all version-controlled projects)
  check-urls    compared installed repository urls agains current locations
  clean-links   remove broken symlinks in systems/
  update-missing  
                download only projects not present yet

  buildsbcl [XC_HOST]  
                download/update and compile SBCL
  world         buildsbcl && build

  clean         delete all compiled object files
  mrproper      delete all downloaded source and fasl files

  slime         run the Superior Lisp Interaction Mode in a fresh Emacs
  lisp          run Lisp in the terminal
  sbcl          alias for "lisp" (with all systems available to REQUIRE)
  openmcl       alias for "lisp"

  listener      run the McCLIM listener
  gsharp        run the Gsharp score editor 
  climacs       run the Climacs text editor
  closure [HOME_PAGE_URL]   
                run the CLOSURE web browser
                (required Debian packages: gif2png,libjpeg-progs)
  beirc         run the Beirc IRC client
  climplayer    run the CLIMPlayer music player
                (required Debian packages: mplayer, fileschanged, fam)
  demodemo      run some random CLIM examples
  clim-alerts   run CLIM alerts
  eclipse [DPY] run the eclipse window manager

  hunchentoot   run the Hunchentoot web server test
  webdav DIR    run the CL-WEBDAV server, serving directory DIR
                (required Debian packages: libssl-dev)

  parse-xml FILENAME
                check XML for well-formedness
  validate-xml FILENAME
                check XML for validity
  validate-relax-ng [--compact yes] XML-FILENAME SCHEMA-FILENAME
                check XML for validity against a Relax NG Schema
  html-to-xhtml HTML-FILENAME OUTPUT-FILENAME
  xhtml-to-html XML-FILENAME OUTPUT-FILENAME
                convert between HTML 4 and XHTMl

  dialog        show an interactive menu 

Maintainance commands for developers:

  record-dependencies
                rebuild dependency information file

If you do 'world' or 'buildsbcl' then SBCL will be installed in
target/ and used for future commands. If you don't run these commands
(or you remove target/) then clbuild uses the 'sbcl' in your PATH.

For configuration options (including for non-SBCL lisps), see clbuild.conf.

It may seem more complicated than it actually is. In reality, clbuild is very easy to use. Say you're developing a web-based application built with HUNCHENTOOT using SBCL, Emacs, and SLIME. Periodically, you want to refresh everything from the development versions of all the packages. To setup your work environment initially, you might do the following:

Install clbuild and make certain you have all the required utilities (see here for details):

darcs get http://common-lisp.net/project/clbuild/clbuild
cd clbuild
chmod +x chbuild
./clbuild check

Download and build the latest version of SBCL (assuming you have a binary SBCL already installed to "bootstrap" the build):
```
./clbuild buildsbcl
```
Download HUNCHENTOOT and SLIME and all dependent packages:
```
./clbuild build hunchentoot
./clbuild build slime
```
Answer "y" when it asks whether to "include dependencies in update? (Y/n)" when HUNCHENTOOT is being built. Then clbuild will download and install SLIME plus the following packages that are used by HUNCHENTOOT: hunchentoot, cl-who, chunga, cl-base64, cl-ppcre, trivial-features, alexandria, babel, cffi, trivial-gray-streams, flexi-streams, cl+ssl, md5, rfc2388, url-rewrite.

Save an SBCL image with all the HUNCHENTOOT and SLIME packages:
```
./clbuild dumpcore
```
Since we saved the SBCL image with the SWANK components of SLIME in it, we need a little "stub" lisp file that will prevent SLIME from trying to re-load SWANK when SLIME is started in Emacs (I called mine "bc-swank-loader.lisp"):
```
(if (not (find-package 'swank-loader))
    (let ((slime-dir (merge-pathnames "lisp/clbuild/source/slime/" (user-homedir-pathname))))
      (load (merge-pathnames "swank-loader" slime-dir))))
```
Then, we need to tell SLIME to load that "stub" instead of swank.loader.lisp on startup, so we put the following in our .emacs file:
```
(setq slime-backend "/path/to/bc-swank-loader.lisp")
```
Of course, there are other things that need to be added to your .emacs file for SLIME as well (for example, you need to make certain that you point to the new SBCL executable and core image), but I won't cover that here (have a look at my earlier post for some SLIME setup tips).

That's it! Now, when you fire up SLIME in Emacs, SBCL starts up virtually instantaneously and all the HUNCHENTOOT-related dependencies are already loaded. When you subsequently want to refresh the packages, you just repeat step #3 above and clbuild will download any changes and add them into your SBCL image. When you want to get a newer version of SBCL, you just do a "./clbuild clean" to get rid of the old fasl's and repeat steps #2-#4 above.

Update-2008-01-17: On #lisp, Andreas Fuchs and Tobias C. Rittweiler made a few more observations:

<antifuchs> billc: afaict, the bigger problem with asdf-install is that not all latest release
	    tarballs work well with all other latest release tarballs; also, that they tend to be
	    quite old and that people downloading them will be left with software they don't bother
	    to hack because it's already outdated anyway
<antifuchs> oh, you wrote that about starter-pack already				      [14:09]
<antifuchs> also, if anything, clbuild is the larger security problem because it doesn't even check
	    gpg signatures (-:								      [14:10]
<antifuchs> other than that, excellent text (:						      [14:11]
<billc> antifuchs: thanks, do you think there are any other usage scenarios that would be useful to
	cover?
<antifuchs> billc: one that tcr mentioned is hacking on the packages themselves, then doing "clbuild
	    diff" to see what was modified locally					      [14:13]
<antifuchs> another is "clbuild update <project>" which pulls the newest version of a project and
	    its deps
<billc> antifuchs: hmm, I saw that but haven't tried it out. looks like it provides a unified diff
	regardless of the vcs - that *would* bue useful
<antifuchs> billc: I think for darcs it provides darcs whatsnew output... but still, it's useful to
	    see what the local state of that lisp world is				      [14:15]
<billc> antifuchs: i liked how clbuild would automatically update the monster.core file after
	updates
<antifuchs> right, that's great (:
<antifuchs> you immediately know if your lisp world is viable (:
<tcr> antifuchs: What happens when I've locally hacked something, and run update.	      [14:16]
<antifuchs> tcr: with VCS things, the regular merging of stuff.
<antifuchs> with tarballs, I'm not sure
<antifuchs> lichtblau will know ((:
<antifuchs> but I believe it overwrites
<antifuchs> (which is why tarballs are evil and VCSed software is nice)
<tcr> antifuchs: What if conflicts arise?
<antifuchs> tcr: then your lisp world isn't viable anymore ((:
<antifuchs> I haven't had that happen to me yet (I think)
<tcr> antifuchs: Is there a way to say "ditch local changes", and "undo last update"? I think that's
      the two possibilities I'd decide between.
<antifuchs> tcr: not yet (:								      [14:19]
<tcr> antifuchs: What I'd also like is a changelog commando, to see if updating is worth it.
<antifuchs> that's a fine idea

January 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31
Dec Feb