Entrepreneur. Creator of Groovy++ Alex is a DZone Zone Leader and has posted 31 posts at DZone. You can read more from them at their website. View Full User Profile

On static compilation of Groovy

11.10.2009
| 16551 views |
  • submit to reddit
Groovy is the great programming language. Period. I am not going to argue about that and I don't plan to convince anybody. If you don't share this belief or need more arguments, please visit the very friendly community at http://groovy.codehaus.org.

I want to talk today about two weakness of Groovy and possible solutions for these problems.

The following "manifesto" starts the main page of the Groovy web site. Please read it:

Groovy...

  • is an agile and dynamic language for the Java Virtual Machine

  • builds upon the strengths of Java but has additional power features inspired by languages like Python, Ruby and Smalltalk

  • makes modern programming features available to Java developers with almost-zero learning curve

  • supports Domain-Specific Languages and other compact syntax so your code becomes easy to read and maintain

  • makes writing shell and build scripts easy with its powerful processing primitives, OO abilities and an Ant DSL

  • increases developer productivity by reducing scaffolding code when developing web, GUI, database or console applications

  • simplifies testing by supporting unit testing and mocking out-of-the-box

  • seamlessly integrates with all existing Java objects and libraries

  • compiles straight to Java bytecode so you can use it anywhere you can use Java


All that is true. I can put my signature on each and every word here and as one of the Groovy Core developers and the Co-Founder of the first ever Groovy company G2One, Inc. (now part of SpringSource and a division of VmWare), I put a lot of effort into making it happen.

What this manifesto doesn't say, and is very important to note, is, as all Groovy developers know, there is a price to pay

  • there is no compile time check
  • there is 5 to 15 times performance slowdown compared to the code written in Java or Scala


In general, the equation we have is:

Great features (including dynamic ones) => Slow and without compile time check


Someone will probably argue that this is not a problem, that the productivity gain which comes out of using Groovy compensates for the disadvantages above, that Groovy became much faster compared to what it used to be, that you can use Java for critical parts of your code, etc.

I don't want to go into this discussion (especially as the one responsible for a big part of performance gain). I think it is almost counter-productive. I only want to mention that, for example, due to performance reasons Groovy is almost useless today for multi-core programming (even with brillian GPars library - disclosure: I am one of developers of GPars).

What I want ot talk about is:

Should it be any price to pay at all?

And if there is such price then what and for what should we pay?


These very important questions have been asked many times in the Groovy community. Though I am not sure if their answers, which were also verbalized several times by several people, were heard. So, now we come to maybe the most interesting part of the story.

Why is Groovy slow and unable to be staticly compiled. Well, because every call is dynamically dispatched.

Ok, totally understandable... But why? Because the beautiful runtime metaprogramming capabilities of Groovy can change behavior of any method (formally speaking, every call on on every call site) Technically, it means that if we found (and knew it for sure) that at some point we call existing method of a class with rightly typed arguments we still can not use bytecode calling this method and even can't be sure that returned value will be of expected type.

Think about it for a second:

Runtime metaprogramming requires dynamic dispatch

  • Dynamic dispatch does not allow compile time checks

  • Dynamic dispatch is slow


All together, we pay a huge price for only one (a very important one) feature of the language. But still... for only one feature.

Imagine for a second that we don't use run-time metaprogramming in some piece of code and we have a keyword or annotation to mark such a piece (method or class or whole module or file extension) to be statically compiled. Does it solve our problem?

I claim it does:

  • It is possible to write staticly typed compiler for Groovy keeping all goodies of the beutiful language

  • It is possible to implement very powerful type inference of local variable and closures, so we don't need to provide verbose type information more than necessary and/or useful for documentation purposes

  • It is possibly to have compile-time metagrogramming (properies, categories, default groovy methods, mixins, ast transformations - all but runtime meta-programmings)

  • It is possible to support Scala-like traits (also knows as interfaces with default implementation), which is extremly strong meta-programming tool

  • It is even possibly to compile piece of code in so called mixed mode (resolve and call statically what we can and call the rest dynamically). This mode is great for example for mixing computation with building markup or populating UI elements with results of computation

  • It is even possibly that because of type inference staticly compiled code can be a little bit less verbose compare to Groovy (almost no need in 'as' conversion for example)

 
There are several reasons I claim so:

  • It is not rocket science
  • Scala proved solution for type inference part
  • There is some work in progress proving each and every of claims above


Let me share with you some  snippets of commented code with you:

/**
Trait Function1 defines interface of abstract function with one argument.

It also additionally provide implementation of several useful methods
Traits are always statically compiled
*/
@Trait
abstract class Function1<T,R> {

/**
Body of the function
*/
abstract R apply (T param)

/**
Creates another function applying another function provided as argument to result of calculation of this one
*/
public <R1> Function1<T,R1> andThen (Function1<R,R1> g) {
/*
here is type inference comes in to play
Groovy allows us not to specify return explicitly AND
compiler knows that return type is Function1 (interface with one abstract method
'apply' because the rest methods have default implementation),
so it can compile the whole new class and return of instance of this class

It is very interesting to notice that we don't even need
to specify type of parameter 'arg' compiler is smart enough to deduct it

Another interesting thing to notice is use of [] operator. As we define
default implementation of getAt method (Groovy convention for []) we can use it
And the last but also important note: In pure Groovy we would write {...} as Function1. Static compiler has enough information to allow us to skip 'as'
*/
{ arg -> g[ apply(arg) ] }
}

/**
Creates another function applying this one to result of another function
provided as argument
*/
public <T1> Function1<T1,R> composeWith (Function1<T1,T> g) {
/*
very similar to previous method
*/
{ arg -> apply(g [arg] }
}

/**
Provides [] - syntax for Function1
*/
R getAt (T arg) {
apply(arg)
}
}


Now we can try to use this code:

    /*
Method to convert iterator to another iterator applying given function to every element
*/
static <T,R> Iterator<R> map (Iterator<T> self, Function1<T,R> op ) {
[ next: { op[self.next()] }, hasNext: { self.hasNext() }, remove: { self.remove() } ]
}


This code looks probably too short. It's done intentionally to suggest to the reader to imagine typical Java code to achieve exactly the same result. Here is the same code with comments:

   /*
Method to conver iterator to another iterator applying given function to every element
*/
static <T,R> Iterator<R> map (Iterator<T> self, Function1<T,R> op ) {
/*
Again as above compiler has enough information to understand that provided map expression should be compiled in to instantiation of a new class
*/
[
// no need to specify return types of methods because compiler knows
// also compiler knows that next () accep no arguments, so correct method will be created
// op [] again at our service
next: { op[self.next()] },

hasNext: { self.hasNext() },

remove: { self.remove() }
]
}


OK, we are almost there. Let us test what we achieved:

        /*
we don't define type of iter variable
isn't it obvious that it is Iterator<String>?

Look:
[0, 1, 2] is ArrayList<Integer>
[0, 1, 2].iterator () is Iterator<Integer>
map {...} called with Function1<Integer,String> (String is return type of provided closure as well as 'it' parameter is Integer)
*/
def iter = [0, 1, 2].iterator().map { (it + 5).toString() }
def res = []
// we use normal Groovy Truth but staticly compiled. So iter instead of iter.hasNext ()
while (iter) {
// and yes, toUpperCase method is at our service
res << iter.next ().toUpperCase()
}
assertEquals (["0", "1", "2"], res )


Let me stop here. There are a lot more things to say about the unlimited opportunities that static compilation of Groovy opens. I will probably blog about it in the future if my research progresses well.

I strongly believe that Groovy can become the main-stream programming language used both in performance-critical and complicated concurrent applications and utilizing both the power of compile time type checking and dynamic genesis of Groovy. Here is my conclusion:

Optional static compilation for Groovy can make the world groovier!

Let us make it happen

Published at DZone with permission of its author, Alex Tkachman.

Comments

Lari Hotari replied on Mon, 2009/11/09 - 3:24pm

 Read Jochen Theodorou's blog article "Fast mode" for Groovy (posted over 3 years ago: http://blackdragsview.blogspot.com/2006/09/groovy-on-speed-fast-mode-for-groovy.html

Lari Hotari replied on Mon, 2009/11/09 - 3:39pm

I would also check out GJIT and the work done by Chanwit Kaewkasi, http://chanwit.blogspot.com/search/label/Groovy . He's got some interesting ideas in his blog.

 

Alex Tkachman replied on Mon, 2009/11/09 - 3:41pm in response to: Lari Hotari

Thank you, Lari.

 I am very well aware about both efforts

Lari Hotari replied on Tue, 2009/11/10 - 6:29pm

There's a fresh article about the same subject in Blackdrag's View .

Charles Oliver ... replied on Tue, 2009/11/10 - 8:34pm

I don't see how this is any better than normal dynamic invocation. You're doing a hash lookup for every call to the iterator-like map, which is basically the same thing as a dynamic call. The only thing this accomplishes is reducing the size of the hash/map you might hit to look them up and possibly reducing the amount of Java reflection code you call through (though that seems unlikely as well). In fact, it seems like this might be even slower, since those hash hits to the iterator-like map won't get inline cached. Maybe I'm missing something.

Chris Ainsley replied on Tue, 2009/11/10 - 11:02pm

Please don't interpret this as Groovy bashing as I really like Groovy as a language, but recently I experimented with embedding Groovy in my own application for performing mapping rules in a tight loop.

I noticed two things.

Firstly that Groovy was noticably slower than Rhino for the same mapping type, and secondly that if I use the  _shell.evaluate(...) method around half a million times, the JVM runs out of memory, even when not storing the results of the evaluation (seems like a memory leak). Rhino was fine for any number of iterations.

 Does anyone else have any experiences with regressive memory use when running the  _shell.evaluate(...) method in a large loop?

Mike P(Okidoky) replied on Wed, 2009/11/11 - 12:26am

Can't we just stick to Scala and be done with it? Seriously, *must* we try to fix a lesser language if the better one is already here?

Alex Tkachman replied on Wed, 2009/11/11 - 1:11am in response to: Charles Oliver Nutter

Charles, I am sorry but I don't understand what hash lookup are you talking about - there is no any lookup at all. Example in the article is standard functional algorithm

Mark Haniford replied on Wed, 2009/11/11 - 1:32am in response to: Mike P(Okidoky)

Can't we just stick to Scala and be done with it? Seriously, *must* we try to fix a lesser language if the better one is already here?

 

No we can't Mike.  Sorry.  It's time for static Groovy

Lari Hotari replied on Wed, 2009/11/11 - 6:33am in response to: Chris Ainsley

Answer to takapa's shell.evaluate(..) question: This is off subject, but your question is handled in this blog article. There also Eval utility class in Groovy API.

Mike P(Okidoky) replied on Wed, 2009/11/11 - 12:59pm

Would it be possible to have the runtime compile a static version of a method while it's running, and have it keep a list of things that would invalidate the compiled static version. Then when somewhere it does something that's on the list, it would mark the method as invalid, and would run the dynamic version next time, which in turn compiles up a static version again.

Second, so away with this big decimal thing and use primitives.

What do the creators of Groovy think about all this, are they willing to change the language a bit?

Chris Ainsley replied on Sat, 2010/06/05 - 5:54am in response to: Lari Hotari

Thanks for the link.

In the end I figured it out and precompiled a number of Script instances but thanks for the link to the article.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.