Clementson's Blog

Debugging Approaches in CL

Monday, June 28, 2004

Peter Seibel asked on c.l.l. for lispers to share what debugging features they find most valuable in their CL implementation. I don't think he got the sort of replies that he was after; however, since debugging Lisp is an area that many new Lispers find a bit, er, unusual (compared to many other programming languages), I decided to summarize some of the replies that talked about how different people approach debugging in Lisp.

Kenny Tilton talked about how he does debugging ( here here, and here):

"Part of the problem for me was that it was too big a question, and would have taken too long to write. Specifics are always easier to answer. As for the question...

• I do not like interactive debuggers. Too slow. Even on Vax Basic I used print statements, even though each new one meant a rebuild and rerun.
• Lisp gives error messages which are an order of magnitude more helpful than the faults reported by C. On dopey bugs, often that is enough. I might need to see the backtrace to find out where the dopeyness started.
• I do not use many local variables, so there are not many to inspect. But I do use OO a lot, meaning I often need to know what's inside an instance passed as a function argument. So it is great being able to double-click something listed in the backtrace to inspect it.
• source debugging? well, control-alt-period in the backtrace jumps to the source of a function, so I kinda have that, tho I get no indication what line I am on. Rarely is that an issue, because other clues make clear where I was (and the functional style helps).
• returning an inspected object to the repl so I can beat on it with my own code is useful, tho rarely needed.
• I like restarting from arbitrary stack frames after hopefully fixing a bug. Some bugs take too long to set up.
• as per another thread, the big win is being integrated with the editor, the browsers, and the running application all at once. To a degree i do not even know what the debugger is, since really it is all Just Lisp to me.

I think we will discover that IDEs aside, lisp itself is so integrated and introspective that no one even thinks of the debugger (the backtrace, right?) as anything separate, nor do they do much debugging from there.

For example, while working on visual-apropos just now I tried doing some fancy-schmancy format string stuff (a first for me). I cocked it up pretty badly, so my grid simply disappeared. Oh, and some unpleasant, inscrutable complaints came back from Tk. So I turned on the debugging print statements and ran it again to see what I was sending. (No, I never test anything, I just move it to production and turn on my pager.) i was sending unrecognizable crap. F1 on 'format' to get to the hyperspec, and now I start playing with format interactively, and learn not one but two ways to do recursion. The second one DWIM, and away I went.

Instead of the print statement I could have set a breakpoint on TK-SEND and then inspected the message, but dozens of messages would have to be manually skipped to get to the grid message. Much easier to just scan the repl window for the bad message.

But backtrace specifically, integration with the rest of the introspective tools offered by AllegroCL. I recall being very happy with the same in MCL. ie, I get out of the backtrace and into the code straightaway, but I leave the backtrace active in case my code investigation raises a question as to a particular argument or the state of some slot.

I think we have to talk about interesting bugs if we want to talk about debugging tools, because the dopey ones need little more than the runtime error and the call stack to debug. With hard bugs i see and undertstand the runtime error, I just cannot make out how my code managed to get there. Based on my understanding of the code and the error, I can sprinkle three print statements each displaying three variables as fast as you can set a breakpoint, step past a few OK calls, and then inspect one variable in one place. So I am ahead nine to one.

But then of course you write pretty simple code, so an interactive debugger is usable. I write table-driven intelligent metasystems which have fewer functions, getting diversity from the DNA of data. Kinda like interpreters. So there is no way I can set a break point on a function, because it will stop there a thousand times before my buggy call is reached. (Now I am ahead nine thousand to one.) But with print statements all those just scroll away until I get back to the same backtrace, at which point I get to scan dozens of lines of diagnostic info to see if I can spot the problem.

With hard bugs it often turns out everything looks good. No problemo, add another couple of print statements and rerun. And with something like cells, I can leave those print statements behind albeit disabled if I sense that they will be useful in the future.

Interactive debugging only lets you analyze one point in time at a time. A slew of print statements covers a long sequence of events, making apparent at a glance where things started to go wrong.

One thing I do to manage excessive output is give my debug print macro some smarts. If the first argument is not a string, it gets passed to a debugp function which by default returns nil. That suppresses any output. I then write a debug-p specialized on the type of the first argument and in that method I can check as many things as I like to really cut down the debug output.

Aside: in all this I forgot to mention trace! Partly because I do not use it much anymore. Not sure why. Maybe because of the metacoding: most of my code is in lambdas."

"When I was programming in C in the late 80's, I evolved a 'no debugging' programming style. I had written some powerful IO routines, that I called my test scaffold. If I had a hairy function to write, I would sit at my terminal typing in two files, hairy-driver.c and hairy-guts.c. Hairy-driver would wrap my IO routines around calls to my code in hairy-guts, giving me a stand alone program. Prior to coding, I would prepare test cases, in the formats used by my test scaffold routines, and write a shell script to do regression testing.

Once hairy-guts.c gets to the point of compiling, I start running my regression tests. They start with null cases, simple cases, and work up. With a preprogrammed set of test cases to work through, it would generally be obvious what the problem was at each stage, without debugging.

So later I'm working on big-prog.c, which uses hairy-guts.c, furry.guts.c, tricky-bit.c, etc. But I have stand-alone versions of the difficult code, so I've already prototyped big-prog as a shell script that strings together various functions. In particular, I can generate my test cases for big-prog using my stand along versions of the components as tools. So I'm working my way through my test cases for big-prog, and perhaps it is not clear where the bug is. Well, I've got the files that my shell script used in order to stitch together the prototype from my stand along components. I don't add printf's to big-prog. I add calls to my IO routines, and get print outs of the internal data structures that ought to match the intermediate files created in the generation of the test cases.

It was a no-debugging programming style in that I only ever had shallow bugs, within the reach of printf and my IO routines. I never had to do deep debugging, delving into tangled code with gdb and the like, so I never did get round to learning any debugging tools.

There were two very different reasons while I developed this plodding, methodical style, one technical, one spiritual.

Technical
Most of my code was signal processing code for the front end of a big speech recognition project. The back end used statistical methods to recognise the speech, based on the reduced dimension description provided by the signal processing. The back end had no a priori knowledge of the correct results from the front end. It took the results from processing the training data as definitive. If the signal processing code was buggy, the back end would simply learn a buggy version of speech. There would be no warning of trouble. One could waste years of one's life doing worthless research, and never suspect.

Since I wasn't going to be receiving bug reports, I needed a coding style that did not depend on them.

Spiritual
Around this time I got involved with a small religious sect, the Friends of the Western Buddhist Order. I enjoyed bug hunting (using print statements). I enjoyed the cycles of elation and despair as the last bug is found and it turns out that it is not infact the last bug. Meditating and studying scripture, I realised that I had cause and effect backwards. Were the bugs the cause of the cycles of elation and despair? No. It was the other way round. I craved the cycles of elation and despair, and that was why I wrote code with deeply hidden bugs in it. If I couldn't escape worldly samsara, I could at least escape coding samara, and move to a coding style with no bugs and no debugging.

Common Lisp
CL supports this coding style wonderfully well. My test scaffold of IO routines is built in with (read) and (print). I don't have to write stand-alone versions of my subroutines, Lisp functions can be run at the REPL, immediately, without even compiling them. The whole business of stitching together the stand alone routines with shell scripts for generating test cases and regression testing is subsumed by the REPL. Working in the language environment instead of at the UNIX prompt strips out a whole level of software development complexity. I've failed to learn the C debugging tools. What could prompt me to learn the CL debugging tools? Ambition. I like to think that my C code was fairly sophisticated from the points of view of phonetics and applied statistics, but it was all pretty basic from the point of view of computer science. I hope to write more ambitious programs in CL, and go beyond the limits of my no-debugging programming style. However macros change the debugging landscape.

Macros
Macros let you abstract control structures. For example, with a list (a b c d) you might want to compute
(f a b),(f b c),(f c d)
Another popular pattern is
(f a b),(f b c),(f c d)(f d a)
It is natural to define macros

* (dolinks (x y '(a b c d ) 'done)(print (cons x y)))
(A . B) 
(B . C) 
(C . D) 
DONE

* (docycle (x y '(a b c d ) 'done)(print (cons x y)))
(A . B) 
(B . C) 
(C . D) 
(D . A) 
DONE

The alternative is to come up with cliches and use them repeatedly in ones code. I've not done too well at coming up with cliches taught enough to bear repeated typing; best so far:

* (loop with list = '(a b c d)
 for x = (car list) then y
 and y in (cdr list)
 do (print (cons x y)))
(A . B) 
(B . C) 
(C . D) 
NIL

* (loop for (x . rest) on '(a b c d)
 for y = (car rest)
 when (null rest) do (setf y 'a)
 do (print (cons x y)))
(A . B) 
(B . C) 
(C . D) 
(D . A) 
NIL

Obviously, mistakes in the cliches result in bugs, and writing a macro eliminates the bugs by design. On the other hand my macros for dolinks and docycles are fairly chunky.

(defmacro with-gensyms((&rest syms) &body code)
  `(let ,(mapcar (lambda(sym)(list sym '(gensym)))
   syms)
     ,@code))

(defmacro dolinks ((a b list &optional value) &body code)
  (with-gensyms(evaluated-list tail)
    `(let ((,evaluated-list ,list))
       (do ((,a (car ,evaluated-list) ,b)
     (,b (cadr ,evaluated-list) (cadr ,tail))
     (,tail (cdr ,evaluated-list)
     (cdr ,tail)))
    ((endp ,tail) ,value)
    ,@code))))
  
(defmacro docycle((a b list &optional value) &body code)
  (with-gensyms (evaluated-list counting-list working-list)
    `(let ((,evaluated-list ,list))
       (do ((,counting-list ,evaluated-list
       (cdr ,counting-list))
     (,working-list (cddr ,evaluated-list)
       (cdr ,working-list))
     (,a (car ,evaluated-list) ,b)
     (,b (if (cdr ,evaluated-list)
      (cadr ,evaluated-list)
    (car ,evaluated-list))
  (if ,working-list
      (car ,working-list)
    (car ,evaluated-list))))
    ((endp ,counting-list) ,value)
  ,@code))))

The issue is: how did I debug that code? My recollections of coding in C are that one dreads having to code up a sophisticated control structure. The cause of the dread is the interaction between the control structure and the code being controlled. What is so dreadful is the issue of controllability. One wants test cases: input data that stresses the control structure by exercising its fancy features and pokes about in the corner cases. With real code being controled it can be hard to contrive the test case to stress test the control structure and shake out the bugs. One might have to come back to it in six months time, when the test case one never managed to construct turns up in live data and exposes a bug. That is when you need fancy debugging tool. That is what I mean by /deep/ debugging.

No such problem arises with a macro which abstracts a control structure. One simply debugs it with bodies that are designing purely for debugging, such as (print (cons x y)). Once the macro is right, one just plugs in the code one wants controlled, and it works. Well, it does if one has taken advantage of the freedom to stress test the macro.

So my theory is that if I use macros to abstract difficult control structures I can write more ambitious programs than I used to write in C, and still not need to learn the tools for deep debugging."

"I did the same but in a different way. Instead of developing testing tools, I adopted a programming style, learned over many years and after many marathon debugging sessions, that lends itself to producing bugless code:

1. Write code in tiny pieces.
2. Never write anything that cannot be immediately tested.
3. Design for the most general case possible.
4. At any stage of development - the accumulated code works. Even if it does little, the little it does is correct!

1. and 2. are easy in Lisp and Forth, but they are possible in C and others as well.

3. is indispensible because code will always change; changes cause bugs; and generalized code is much easier to modify than code specific to one solution.

4. insures that bugs do not creep in and become so deeply embedded they are hard to find and get rid of.

This style clearly involves lots of testing but the tesing is done on small chunks which makes bugs easy to detect and easy to fix.

This style takes more time up front (time during which managers may get nervous because nothing visible is happening and others have written mounds of untested code) but the time saved at the other end is so enormous the manager will be kissing your hand while cursing the others who are slaving over their debuggers."