Working with a Metadata-based DSL in Emacs
Monday, October 15, 2007
When you create a Domain-Specific Language (or DSL), there are a variety of different approaches that you can take. I'll be a bit simplistic and talk about just 2 of these (which I'll loosely categorize as the "source approach" and the "metadata approach")
- The "source approach" is normally to just create a source-based mini-language using a "general purpose language" (GPL) like C or Lisp. Typically, when done with Lisp, one builds the mini-language "on lisp". This has the advantage of allowing you to "take Lisp with you" - both for working with programs that are written in the DSL as well as when you subsequently might want to extend the DSL. It also means that you can usually use your same editor (yay Emacs!) to write programs in the DSL language. When done with non-Lisp languages, this approach still results in a DSL that you can frequently use standard tools with; however, you don't usually have the power of your GPL available for use. (Note: I've posted a number of times about Lisp-based DSL's in the past. In particular, "The Lisp Difference" and "Developing DSL's in Lisp" have been popular).
- An alternative approach (and one that was adopted by many of the "business rules" companies during the 90's) is to create a metadata-based language, storing the logic in database tables (or some alternative, non-source code, format). With this "metadata approach", you have tremendous freedom to change the overall "look and feel" of the DSL (although, in practice, this is rarely done) but it also means that you usually have to write your own tools for working with the language.
A lot of the contract work that I do is for companies that are running the J.D. Edwards (JDE) ERP software package called OneWorld (also known as "EnterpriseOne" and known originally by the "internal" code name "Everest"). JDE used to be an independent company before they were swallowed up (friendly acquisition) by PeopleSoft and then gobbled up (hostile acquisition) by Oracle. Oracle still sells and supports the software; however, they also sell the ex-PeopleSoft ERP package and Oracle's own ERP package as well as the JDE ERP package, so there's quite a strong demand for independent developers who know the software. I definitely fall into that category as I used to be one of JDE's senior architects (starting off as a contractor working for JDE, accepting a senior development role with JDE, creating the integration methodology for OneWorld XPI, staying on as a cross-platform architect after the PeopleSoft acquisition, and leaving following the Oracle acquisition). So, even though I like to do Lisp-related things, I still do a lot of JDE stuff because I have so much JDE development knowledge. In addition, I sometimes get to use Lisp in some of the JDE-related tasks and this can prove a competitive advantage (unfortunately, when I do use Lisp in this context, it is usually for some task that I can't blog about). ;-)
Anytime you're involved with a particular piece of software for any length of time, you tend to build up your own custom set of tools and development practices. Most of the JDE logic is coded in a DSL called EventRules (ER). This language is stored in a custom metadata format (originally a proprietary database format known as "TAM specs"; but, in more recent versions of the software, the metadata has been stored as XML). Here's a look at how a table record fetch (in this example, fetching a record from a table called "F060116") appears in it's metadata (TAM spec) representation:
The metadata on the left-hand side is just a higher-level "index" (from the calling application metadata to the ER logic metadata) into the lower-level metadata ER code that is shown on the right-hand side. The grid on the right shows that the table's name and some (not easily understandable) additional characteristics (for the F060116 table record fetch) are contained in the metadata in the first tab while the 4 parameters for the fetch are contained in the other 4 tabs (you can see the tabs but the contents are not visible in the picture). Since the internal representation is metadata and not source code, the only way most people have to work with it is through a custom JDE ER editor that "translates" the metadata into a source code-like representation. As with most custom one-off editors, this one is a piece of s..., er, I mean, this one is sub-optimal. ;-) In order to view parameters and calls, you have to drill down into each call using a separate editor component. Navigation through the custom editors is time-consuming and there are limits to the types of cross-referencing, VC, and other code management tools that are provided by the OneWorld toolset; however, the biggest hassle is just the limited functionality of the custom editors. This makes ER logic development quite tedious. Here's an example showing the table fetch from the previous example. In the ER editor, the single line that does the table record fetch is double-clicked and, in the popup window, the programmer has to scroll down through the list of fields in the table in order to view/change the "work" fields that get mapped to the table fields for the fetch:
As you can see, this makes it hard to understand the code since you have to drill down into each line of source code in order to modify it or even to fully understand what the line does (since, if the operation on a given line takes parameters, the parameters won't be visible until you do the drill down). In order to make it easier to work with ER code, I've got a utility that will output the ER specs in a viewable format to a text file. This makes it a lot easier to build custom xref's and to search through the code. I use Emacs a lot when working with ER code and have created a OneWorld Emacs major mode so that I have some basic code colorization of key syntax components in ER and some "convenience" functionality so that I can easily switch back and forth between the OneWorld editors and Emacs. Since the exported ER specs contain expanded parameter lists, this makes it a lot easier to understand and work with ER logic. Here's the same example snippet of code in Emacs:
By looking at the code in Emacs, I can tell that the F060116 fetch uses index#1, that it uses "AddressNumber" as a key, and that it returns 3 fields from the table into form-level variables (a character variable, a string variable, and a date variable). These are all things that I wouldn't have been able to see in the ER editor without drilling down into the line of code and scrolling through the list of table fields (and, even then, I would have had to call out to another custom editor to retrieve the field-level information). There is also custom "colorization" of ER syntax, making it easier to read the code. I can also select a OneWorld object name in a source member in Emacs (for example, by double-clicking on "F060116" in the above code snippet) and press a key sequence to either display that object's source in Emacs or edit that object using the appropriate OneWorld metadata editor. In addition, since I can easily output all source, I can use standard tools for cross referencing, version control, and searching. This makes it an awful lot easier to work with OneWorld objects and to manage large development projects in OneWorld.
My "OneWorld major mode" most certainly won't be of use to many people (the intersection of the population of OneWorld ER developers with Emacs users is really, really small); however, for what it's worth, here it is: ow-mode.el. A quick run-down of what is provided by the major mode:
- Manages both the retrieval of the OneWorld metadata into a source code textual format and the routing of object edit requests to the appropriate OneWorld editor component.
- Creates a few basic key bindings so that one can view ER code from within Emacs and launch the OneWorld editors when necessary.
- Sets up 3 different sets of font-lock colorization:
- Keywords
- Constants
- Variables
ow-font-lock-keywordsvariable. - Defines
ow-mode-syntax-tableso that Emacs knows some basic syntactical constructs in ER code (e.g. - that variable names can contain underscores and that comments begin with "//" in ER code).
So, even though most people won't be able to use the code, I hope this has at least been an example of how it's still possible to "live in Emacs" even when the application you're working on is written in a metadata-based DSL with a set of closed, proprietary tools.
