Saturday, June 23, 2012

ClojureCLR 1.4 released with code gen redone

I just pushed version 1.4.0 of ClojureCLR.

This version matches all fixes and enhancements in the same version of Clojure/JVM/TheMotherShip.  It also  sports, at long last, the new non-DLR-based code generation phase for the compiler.  My earlier post titled "Code gen redo preview".  The benefits outlined in that post--smaller assemblies, faster startup, some speedup--are now available in this release.

In the preview post, I mentioned attempting to re-implement 'light compilation' in the new compiler.  The previous  (DLR-based) version of the compiler has a special compilation mode used during evaluation (but not during AOT-compilation) for any function that does not need to have its own class defined.   It used DynamicMethods, a lighter weight version of IL generation that can be significantly faster than methods generated to assemblies.

The way the DLR generated DynamicMethods allowed live constants to be embedded in a way that full compilation did not; as a result, there were some constructs that would load evaluated but would not AOT-compile.  It never happened to me, but I know of one person who got bit by that (and who asked me to leave light compilation out of this version).  That person need not worry -- I have not yet finished coding light compilation and it is not active in this release.

Does it matter?  Most likely not.  You could see some speedup loading very large files, such as clojure/core.clj.  However, I'm pretty sure light compilation was massively speeding up running the Clojure test suite.  For some reason, with the current version, if you start up ClojureCLR under the debugger and run the full test suite, msvsmon.exe, the Visual Studio Remote Debugging Monitor, goes absolutely nuts allocating memory.  It appears to max at about 6GB on my PC with 8GB.  One sees some serious thrashing at that point.  I have no idea why.  I have no problem compiling the entire clj source bootstrap code, which starts off evaluating core.clj and friends first and then going back to compile it, with not a hiccup from msvsmon.

So, at present, I run the test suite from the command line.  If I need to debug on the tests, I just load the test file in question and run the tests for it only.

If you see this kind of memory-hogging behavior in mvsmon.exe, let me know.

The near-term roadmap for ClojureCLR development, in no particular order, is:

  1. Catch up with Clojure 1.5.x-alpha changes, particularly the new work on reducers.
  2. Examine the perf of the test suite and see what's going on there.
  3. Work on a single assembly distribution of ClojureCLR (using nested assembly loading), following the trail blazed by aaronc and Ralph Moritz.
  4. Work on running ClojureCLR on Mono.  (Some very positive reports from Robert Johnson on his experiments).
  5. (Maybe) finish implementing light compile, for my own sense of fulfillment if nothing else.
The work on reducers on the JVM side has introduced version-specific implementation (using Fork/Join in JVM 7 or falling back to other means on earlier versions).   I suspect a similar 3.5/4.0 .Net distinction could be made to take advantage of the Task Parallel Library (or whatever it is called these days).  If you've played with TPL under ClojureCLR, I'd love to hear about it.