Bill Clementson's Blog

SOAP Web Services in CL

Wednesday, June 23, 2004

Several weeks ago, the Denver Area Lisp Users Group had their May meeting at UCHSC ( Larry Hunter runs the Center for Computational Pharmacology Bioinformatics research lab at UCHSC in Denver and they are the largest Lisp shop in Colorado). At our meeting, we looked at the work they are doing with integrating Web Services technologies into the Bioinformatics work they do (have a look at this ppt presentation for some background on their projects). Gordon Weakliem has already posted about the meeting, but I've been holding off until I could post some of the actual presentation material. I have just received the material from Andy Dolbey (the presenter at our user group meeting) and am posting it here (in zip file and tar file formats). Included in these archives is Andy's presentation (in both Word and PDF formats) and code samples. Andy emphasized that the code in these files is a "work in progress" and should be used for illustrative purposes only. Here are Andy's comments on the material:

"During the presentation, I went through some of the examples that come with Allegro's SOAP package. I'm sending you both of the files that provide that code:

• soapex.cl
• soapval1.cl

In the first one, soapex.cl, I indicated with the following comments the ones I went through:

;; --------------------
;; I SHOWED THIS ONE:

I have left soapval1.cl alone, because I pretty much went through each of the examples in that file.

Code in the soapval1.cl file ties in to efforts I made earlier on to do two things: create a client to access some of the services defined there, and later, create a WSDL descriptor file for clients (both lisp and perl clients) to use to connect to the service. To show a little touch of that work, I've included these files:

• countTheEntities.pl (ties in to ACL SOAP "countTheEntities")
• perlClient-to-lispSrv-5-18-04.pl (ties in to ACL SOAP "manyTypesTest")
• count-entities.wsdl (ties in to ACL SOAP "countTheEntities")

Finally, I'm sending you some material that Dan McGoldrick, a lisp colleague of mine in Larry's lab, has been working very hard on. It's based on a very complex lisp tool Dan has made, called descriptron. The goal of descriptron is to tie in and link up dynamically a large number of related but disparate molecular biology databases. There is a great need for this sort of tool in bioinformatics, and all of the people in our group want access to this tool. Yet quite a few of the people in the group are not comfortable with lisp. This makes it a perfect candidate for WebServici-fication. These are files of some of the code that has come about over the last couple months:

• descriptron-web-services-local-tst.lisp - test version of service, client
• dProxy.cgi - a perl code bridge for getting from java to lisp, due to problems with java
• dtron.wsdl - a descriptor of the service, actually generated by the ACL SOAP API
• descriptron-call-new-wsdl.pl - perl calls descriptron with the wsdl service descriptor

For service clients, the data flow to and from this tool is very complex. From a lisp angle, it involves many lists of lists, sometimes embedded, sometimes not. For SOAP, the data types will in most cases have to be rendered as SOAP structs. For lisp, this is no problem. But it gives rise to very tricky and stubborn complications on the perl side. For example, a sequence of structs that descriptron serializes and transmits as an XML message will be received as a list of hashes on the perl side. Sounds reasonable, but the keys of the hashes have the same names, and so the client only ends up with the final one (earlier ones having been repeatedly overwritten). However, had the server sent it across as a SOAP *array* of structs, perl would handle that with no problem at all: the message will be deserialized an array, where the members of the array are anonymous hashes, whose keys are safely hidden away from other members' identically named keys. The problem here, though, was that until recently, Allegro's SOAP API only supported SOAP arrays whose elements are simple types like strings or ints. This has been fixed now, but was a problem for quite a while. It was a problem in two ways:

1. It takes a certain amount of detective work to figure out where things are going wrong.
2. Developers using other languages' SOAP APIs are forced to wait until the lisp API creators make the necessary changes, or instead, not rely on their own APIs' deserialization tools to decode the message, and instead try using raw XML parsers to dig out the pieces of the message.

The elements the lisp server sent are there in the SOAP body, it's just a matter of a client getting to them.

This raises a very important issue for SOAP developers: if I am developing a SOAP service, to what extent should I be concerned about catering to the intricacies, complications, and limitations of all the possible client types that might to try to access my service?? For our group, it's a project that we simply must get to work, and we only have 3 main languages we're using: lisp, perl, and java. With this setting, we are willing to bend over backwards here on all sides, for all of the SOAP client and server development we do across these 3 languages. But in a bigger picture, how much will I (or even can I!) care whether or not a .NET based VB client can or can't access my service? It's a tough issue for the SOAP world right now. The hope is that the W3C will one day work to make SOAP and WSDL actual standards, rather than evolving recommendations, and that language and API designers will then faithfully follow such standards. But we're definitely not there yet.

OK, those are some of my thoughts about our SOAP experiences. To this day, from afar, I put in long hours of work in the evenings and on weekends in order to finally get our project completed. It's been a great learning experience, but also a time sink."