Clementson's Blog

CL's MOP vs Java's Reflection

Monday, September 19, 2005

When encountering the MetaObject Protocol (MOP) in CL for the first time, people often try to equate it to reflection in Java. However, equating the two does a major disservice to the MOP. On c.l.l., Emre Sevinc brought up the issue and received in reply from Pascal Costanza a very good summary of how the MOP compares to Java reflection. The entire thread can be found here; however, I have provided the core exchange below (Note: I've made some minor edits and added some links. Emre's comments are in red, Pascal's are in black and blue):

"I'm reading The Art of MetaObject Protocol and am trying to understand how it extends CLOS.

I've read about 60 pages and a question came to my mind:

Authors talk about dynamic class creation, inspecting classes at runtime, etc. How are these features compared to "reflection" in Java, C#, etc? I don't know much about that; but, as far as I know, Java and C# programmers can also introspect classes during runtime, create classes on-the-fly, etc. which they call the "reflection" property of their language, right?

Is the MetaObject protocol for CLOS similar to "reflection" in those languages? Is it something more, something less? Does it have some advantages that reflection in Java and C# don't have?

Reflection originally means both introspection and intercession (these are the technical terms). Introspection allows you to find out information about elements of your program - classes, variables, methods, functions, etc. What the Java folks called reflection in the beginning is indeed just introspection.

Intercession is a way to modify the characteristics of the elements of your program. So intercession allows you to change the properties of classes, variables, methods, functions, etc.

Reflection was originally "invented" by Brian Smith in the form of procedural reflection - i.e., you basically have a notion of interceding procedure calls, inspecting the environment(s) that are active during those calls, and possibly changing the way the program continues its execution. If you allow programmers to intercede at each and every single step of your program execution, though, your programs slow down considerably, even if noone intercedes anything, because the runtime has to continually check whether someone wants to do something additionally or not.

For that reason, Brian Smith and Jim des Rivieres (if I understand correctly, it was those two guys) implemented reflection by dividing the program into levels, so that one could specify at which level you want to intercede the program execution. The levels for which you don't intercede can be executed "at full speed", while the other levels have to run your additional code.

It was then discovered that this actually leads to an object-oriented design of the reflection facilities, because the notion of dividing programs into levels and specifying different behavior at different levels is more or less exactly what you do in an object-oriented hierarchy. So, for example, in a class hierarchy, the classes higher up in the hierarchy define the "base" behavior while the classes that inherit from those superclasses define the additional behavior. Next, when you disallow programmers from changing the definitions of the base classes, you can actually implement the runtime much more efficiently because you can check whether a given object is an instance of a base class or not, and if it's an instance of a base class you can run optimized versions of the methods defined on those classes that don't call the (non-existent) code of subclasses.

The CLOS Metaobject Protocol is essentially that idea carried through the end.

Hmm, reading your words ring a few bells, similar to what AMOP authors tell about on-stage objects, backstage objects, etc. And redefining being *backstage*, being able to modify objects that I have created.

...modify _metaobjects_ that you have created. Of course, this ultimately means modifying base-level objects, but when you write meta-level extensions, you are focusing on the meta-level objects (i.e., objects that stand for classes, slots, generic functions, methods, etc.).

Java reflection is not even close. Java has meta-level objects that describe classes, methods, fields, etc., but there is no way to change them. However, there are some little opportunities to get something along the lines of a full-blown metaobject protocol:

• You can implement proxy classes - see java.lang.reflect.InvocationHandler. This allows you to intercept method calls to objects.
• There is a debugging API that allows you to change the definition of a class (including its methods) at runtime, which in principle allows you to add additional behavior to methods.
• Finally, you can intercept class loading and modify the bytecode for a class before it enters the virtual machine. This is the most powerful way to achieve similar effects as those of a MOP, but is not a proper MOP in its own right. (You don't specify the new behavior as methods for meta-level classes, but you rather just modify a byte stream.)

So, you say that in Java it is possible to change a class, add/remove properties, add/remove methods (did I get it right?). I can change the bytecode before it is loaded but this is low-level stuff, not modifying a class using Java constructs themselves, right? And once the class is loaded, I'm limited in some way.(?)

In Java, once a class is loaded, you cannot change its structure anymore. That is, you cannot change the number and types of fields, and you cannot change the number and signatures of methods anymore. If you want to do that, you have to do that at load-time (or compile time, of course). What you can change at runtime is the definition of methods, that's it.

(Sun has a working implementation of the JVM in their research labs in which you can actually change the class structure in more depth, but that's considered dangerous and therefore not released to the public.(!) At least that was the state of affairs a few years ago. Maybe they have added a few things, but I am not keeping track of the JVM anymore.)

The latter has been used to define proper (load-time) metaobject protocols, though, for example Javassist and Reflex.

Are they written in Java, can they be counted as simply "extending" the language mechanism (I assume this can be said for MOP, using simply the CLOS itself to extend itself, but of course I'm nothing but a confused newbie, that's why I ask).

Yes. I think this becomes more useful, btw, with the annotations they have introduced in JDK 5.0 - then it also becomes more clear that it is indeed a language extension mechanism. Annotations are kind of user-defined modifiers, like public/protected/private, static, abstract, transient, volatile, etc. The load-time class modification approaches can take these annotations into account and change the bytecode accordingly, which amounts to providing a language extension facility. Kind of like macros through the backdoors.

These things are much more complicated to achieve in Java than in CLOS, though, even with such frameworks. Furthermore, I am convinced that a full runtime metaobject protocol is essential. (Load-time or compile-time metaobject protocol can lead to more efficient code, that's why they have been pursued in the past, but I think these efficiency gains are too small to be important enough in the long run. The flexibility of a runtime metaobject protocal is more important because it gives you more expressivity IMHO.)

Let me rephrase it again: You say that Java has load-time (just before loading the classes into virtual machine) and/or compile-time metaobject protocol and this contrasts with Common Lisp MOP which lets you play with classes and methods during runtime, right?

Right.

I think the situation for C# is similar to that for Java, although they have added a few more practical stuff from the beginning. For example, .NET comes with a library that gives you a (modifiable) representation of their bytecode format, and I recall seeing a presentation about a load-time bytecode modification approach for .NET that was a lot simpler than what I have seen in the Java world. But that's all I can tell about C#.

The only detailed example of reflection in C# I know of is from the book Programming C#, 2nd Edition by Jesse Liberty and there's a full chapter at the URL:

http://www.oreilly.com/catalog/progcsharp2/chapter/ch18.html

Basically the author states that:

"Reflection is generally used for any of four tasks:

Viewing metadata: This might be used by tools and utilities that wish to display metadata.

Performing type discovery: This allows you to examine the types in an assembly and interact with or instantiate those types. This can be useful in creating custom scripts. For example, you might want to allow your users to interact with your program using a script language, such as JavaScript, or a scripting language you create yourself.

Late binding to methods and properties: This allows the programmer to invoke properties and methods on objects dynamically instantiated based on type discovery. This is also known as dynamic invocation.

Creating types at runtime (Reflection Emit): The ultimate use of reflection is to create new types at runtime and then to use those types to perform tasks. You might do this when a custom class, created at runtime, will run significantly faster than more generic code created at compile time. An example is offered later in this chapter."

In the "reflection emit" section, the author describes a few methods and all of them except the last one includes writing to a file the description of the class and then compiling and loading and using it. And in the last method, he "emits" MSIL opcodes directly (without writing to a file) which leads to creation of a .NET "assembly" whose methods can be invoked. However, I think this is similar to what you said about Java, e.g., creating some kind of bytecode that corresponds to a class and then using it.

Metaobject protocols exist for other languages (and also for other domains than language implementations). A relatively old list of metaobject protocols is given at http://www2.parc.com/csl/groups/sda/projects/mops/existing-mops.html

Eric Tanter provides an excellent and recent overview of reflection, metaobject protocols and aspect-oriented programming (which is another form of reflection, in a sense) in his PhD thesis about Reflex - see http://www.dcc.uchile.cl/~etanter/

Thank you very much for the links, I'll check them out.

By the way, the AMOP is about 14 years old and does it still reflect the MOP in Common Lisp or are there additions, modifications, etc. to MOP?

Yes, it hasn't changed since then. Some suggestions have been made to change parts of the specification. For example, google for "Kiczales" in comp.lang.lisp and comp.lang.clos, and you will find a few discussions. There is also a paper published in the book Advances in Object-Oriented Metalevel Architectures and Reflection, edited by Chris Zimmermann, with some more detailed suggestions. But none of them have caught on.

This is mainly because the balance between giving the programmer of a CLOS implementation more possibilities to optimize things and the user of the MOP more flexilibity to change them is a delicate issue. For this reason, in fact none of the CLOS implementations ever implemented the full specification, as far as I can tell. All of them deviate in some regard or the other. That's why you can find compatibility libraries in several places that try to make up for some of the incompatibilities of CLOS MOP implementations. My own Closer to MOP project is one of those, and I hope to provide a somewhat more general approach than others that typically focus on what is needed for some concrete project.

My overall impression is that the CLOS MOP specification works quite well, and is indeed successfully used in practice. However, there are some things that are underspecified, or not quite right. The AMOP specification itself clearly states that it shouldn't be taken as the final word on that matter. So there is definitely room for improvement. "