Bill Clementson's Blog

Dynamically changing running Lisp code #2

Tuesday, February 24, 2004

Yesterday, I commented on how you can dynamically change running Lisp code and posted an explanation and example of this from Edi Weitz. Rob Warnock has also posted some detail about how he does the same type of thing:

"I'm developing some Lisp-based middleware for a web application -- that is, there's an Apache web server and a PostgreSQL database and a CMUCL image in the middle that gets requests from Apache[1] and pokes around in the database and eventually spits out some HTML via Apache to the browser user. A typical development session has the following windows open[2]:

• A Netscape browser window.
• An "xterm"[3] to the REPL[4] of the middleware.
• One or more editor windows into the application's Lisp code.
• [Sometimes] Another "xterm" open running "psql", the command-line interface to the PostgreSQL database.

Now you might not want to do exactly the following in production, because running the ASDF "LOAD-OP" does add a few milliseconds to the access even if it has nothing to do, but during development the following is terribly convenient:

% cd /usr/local/apache/htdocs/some/path
% grep -i lisp .htaccess
# sock.cgi is the little CGI trampoline that connects to the Lisp process.
Action lisp-handled-pages /sock.cgi
AddHandler lisp-handled-pages .lhp

% cat foo.lhp

   ;;; Boilerplate for using ASDF from LHP page.
   (flet ((this-page (http-request)
	    (asdf:operate 'asdf:load-op "foo")  ; Ensure up-to-date each time.
	    ;; Must explicitly INTERN, since package may not exist before now.
	    (funcall (intern "MAIN" "FOO") http-request)))
     (lhp-set-page-function #'this-page))

%

Don't worry about all that LHP stuff (it's part of the infrastructure of the persistent Lisp server), except to note that the very first time the page gets accessed, that little bit of Lisp code gets LOADed and the function #'THIS-PAGE gets registered as the handler for that URL. Then #'THIS-PAGE will be called, which will compile & load all of the files in the "FOO" subsystem (via the ASDF call), and then finally will call FOO::MAIN (which should "do stuff" based on the data in the HTTP request block and/or the POST data, and emit some HTML).

Now here's the development cycle:

1. Using the Netscape window, access [or later, "Reload"] the page. (If nothing has changed, this will just call FOO::MAIN.)
2. In the REPL window, note the logging of the web request, and the results of the ASDF:OPERATE (if any). In the Netscape window, see if you got the correct HTML results displayed.
3. If there were compile errors, fix them in an editor window, "Save" the file [but stay in the editor], and go back to #1, which will automatically (thanks to ASDF) re-do the compile & load (but of only the sources which changed and the modules that depended on those).
4. If there were logic errors [that is, the output to the browser wasn't correct], you can edit the source, save it, and go back to #1, which will recompile whatever's needed and try the access again.

As far as finding your problem, you can:

1. Use the REPL window to poke at bits at the last web request, e.g.:

   app_srv> (http-request-method *req0*)
   "GET"
   app_srv> (http-request-self *req0*)
   "/~rpw3/foo.lhp"
   app_srv> (http-request-xpath *req0*)
   "/u/rpw3/public_html/foo.lhp"
   app_srv> (let ((h (http-request-handler *req0*))) (uri-handler-function h))
   # <Closure Over ORG.RPW3.CGI.LHP::CACHED-LHP-PAGE-FUNC {48335039}>
   app_srv>

2. You can trace/untrace one or more routines within the app, then hit "Reload" on the browser and watch the call graph flow.
3. You can call the lower-level functions of the application directly from the REPL, and look at the results. [It is helpful to have a wrapper function that will "clone" an existing HTTP-request block and call its handler function, but with output to the REPL stream.]
4. You can hand-code some debugging functions [in another editor window] and LOAD them into the image, then edit the app's code to call them at strategic points.
5. If you're getting unexpected conditions being thrown, you can cause a backtrace to come out on the REPL, and either choose to enter the debugger or not. [The wrapper mentioned in #3 is helpful here, too.]

And so on, the usual stuff.

The point is that the it gives you "fingers" into almost every aspect of the running system. The time around the "try/oops!/debug/fix" loop -- just minutes, sometimes just seconds! -- is VERY much shorter than if you had to do a traditional "make" and restart the whole middleware server each time. I find it tremendously productive!

[And all the while, the part of the web site you're not working on can continue to be accessed without interruption, if you need to do that...]

Oh, and did I mention that all of the HTML gets generated using Lisp macros (using htout)? Displaying the results of a database query in an HTML table is really easy when you can intermingle your Lisp code and HTML using a single syntax (S-exprs) for both:

  (let ((column-names (first results))
        (rows (rest results))
	(stream (http-request-stream request)))
    (with-html-output (s stream t)
      (:table (:border 1 :cellspacing 0 :cellpadding 1) ; make compact
        (:tr ()
          (dolist (name column-names)
            (htm (:th (:nowrap) name))))
        (dolist (row rows)
          (htm (:tr ()
                 (dolist (item row)
                   (htm (:td (:nowrap) (esc item))))))))))

==========
Footnotes:
[1] Not exactly the same as "mod_lisp", but close: a tiny C-coded CGI trampoline connects to the CMUCL image via a Unix-domain socket, then speaks a "mod_lisp"-like protocol to it.

[2] When I was just starting out with using Common Lisp in web apps it was using classical CGI, in which a whole fresh CLISP or CMUCL image would get fork/exec'd for each page hit. In that case, I found it useful to have another "xterm" running "tail -f /usr/local/apache/logs/error_log" to see the error messages in realtime. Now, though, with condition handlers wrapped around the HTTP request handling code, you can see any problems directly on the main REPL window (or in the dribble log "detachtty"[3] provides).

[3] Well, the "xterm" is actually running "attachtty" connected to a "detachtty" that started the Lisp image at boot time, giving REPL access to the detached daemon. Note that accesses to the server also get logged there, e.g.:

; Feb 24 08:29:45.11 cgi-sock[723]: GET "/lhp/clhs"
; Feb 24 08:29:53.69 cgi-sock[724]: GET "/lhp/clhs?state=quick&pat=db"

and that "detachtty" supports a dribble log file, so that you don't miss stuff that happens when you weren't watching. [The script I use to run "attachtty" does a "tail -30" of the dribble file before exec'ing "attachtty", which provides easy access to some recent history.]

[4] The server supports additional REPLs with a "telnet /path/to/.sock.repl", which connects via a different Unix-domain socket to the CMUCL image and starts up another top-level REPL thread, but in practice I've found I don't really use those much -- the single "attachtty/detachtty" is usually enough."