A disruptive technology for code generation

OK, now I have to support my claim. Here we go.

I've been using StringBuffer.append() for code generation (like everyone else) while looking for safer alternatives (safer meaning, compile-time type-checked). I've seen all the model-to-text languages. But the rest of this post is not about them.

One way to ensure syntactic validity when generating source code relies on instantiating EMF classes that stand for Java source elements. Verbosity can be reduced using fluent interfaces instead of directly invoking EMF factories, as reported in http://bugs.eclipse.org/234003

But problems remain. Another (perceived?) problem of AST-to-AST is the laborious coding of visitors, pattern matching and rewriting, multi-stage AST transformations, yes, model transformations. As usual, DSLs have been proposed to solve that, too. They are DSLs, i.e., another language to learn. You want to learn them? This way please.

But LINQ is on the march, and it's being used for code generation. It's being used to query the input (which can be even C# API elements, using LINQ to Reflection) and it's being used to piece together the resulting code:

• http://themechanicalbride.blogspot.com/2007/02/do-you-know-linq-fu.html
  
• http://www.interact-sw.co.uk/iangblog/2007/04/03/linqcodegen
  
• http://msdn.microsoft.com/en-us/library/bb546150.aspx
  

As the ultimate test of suitability for AST-to-AST, LINQ is being used to refactor (refactor!) C# ASTs, no less. Compare with the article Unleashing the Power of Refactoring (and that article does a great job of explaining the JDT API for refactoring, so the comparison is *not* biased toward LINQ)

LINQ is not a DSL, its declarative constructs are based on logical commonalities in data access, better factored than those of model transformation DSLs. To my (informed) taste at least. And you're always free to extend LINQ with operators of your own, in a type-safe way that is. Or perhaps what finally convinced me were all the optimizations performed out-of-the-box when evaluating LINQ:

• indexes on in-memory collections, or
• taking advantage for free from multicores, or
• resultsets automatically refreshed on the wake of updates to their base data.

It's a pity we have no Java-based library to execute LINQ queries, queries expressed as ASTs (remember http://bugs.eclipse.org/234003 ?) although dedicated textual syntax would be nice, too. All right, all right, even if you'll settle only on textual syntax there's an interim solution till Java supports lambda expressions.

There's also talk about "LINQ in Scala". What I say is: it's not about syntax! (that's the easy part in fact) What makes a BIG difference are the optimizations supported by the engine when evaluating LINQ queries, input syntax notwithstanding.

If the above strikes a chord with you, let's join forces! I want to get rid of StringBuffer.append()! One of the grassroots movements to implement LINQ in Java is http://groups.google.de/group/jlinq and with some luck we won't need to wait for Java 7+ to enjoy the benefits of language-integrated queries.

P.S. If you need to convince your university department that the above qualifies for a master thesis, just flash the following papers and they'll accept immediately:

• Phil Wadler and Simon Peyton Jones.
  Comprehensive comprehensions: comprehensions with "Order by" and "Group by"
  


• Darren Willis, David J. Pearce and James Noble.
  Caching and Incrementalization for the Java Query Language.
  

(you could also slip in one of my papers ;-)

• Garcia M., Möller R.
  Incremental evaluation of OCL invariants in the Essential MOF object model
  

Of course there are other opinions on best practices around code generation. But I hope you'll be reminded of LINQ when tracking down some obscure bug due to StringBuffer.append() in your favorite template language :-)

Miguel
http://www.sts.tu-harburg.de/~mi.garcia

Semantic-aware component provisioning: "pencils off" status report

I want to recap my summer of code so far, before the final evaluation starts. Here's what I accomplished so far. It was a lot of work, and a huge fun as well, hope to catch your interest and get some feedback on future developments! :)

GUI CONTRIBUTIONS

• new component submission view (drag and drop support from the repository explorer to add dependencies easily :) )


• repository explorer view


• preference page (set repository URL)

IMPLEMENTED FEATURES

• submit a new component


• usual search features (name, version, tags)


• search components providing a set of tasks


• search components providing all the tasks of the selected ones


• "smart" search of components functionally equivalent to the selected one (reasoning here)


• search components depending on the selected one


• assert functional equivalence between components

HANDS-ON

Let's take a test drive. I submit a new component, in this case (just as an example) the "last-gsoc-demo" one. I fill in some data, and press submit. I can just drag-and-drop dependencies from the repository explorer to the dependencies viewer.



I previously submitted some sample components. Now since all jdbc drivers implement the same specification, to some extent it is correct to consider them "functionally equivalent", and I push this statement in the knowledge base.



For sake of brevity I ask you to trust me without further shots, what I did is just assert all jdbc drivers in the repository (besides the "dummy-jdbc" one) "functionally equivalent" to the postgresql one, and then assert the "dummy- jdbc" only equivalent to "mysql-jdbc". I can ask now the repository to give me components "functionally equivalent" to the selected one ("dummy-jdbc"), just clicking on the context menu item:


Here's what I obtain:



You might notice that the selected item is still there, which makes sense since everything is of course functionally equivalent to itself. ;) Furthermore, it is worthwhile to note I only said the "dummy-jdbc" was equivalent to "mysql-jdbc", full stop! The rest is just the result of the reasoning process.

Now, I can also describe a component in terms of the "tasks" it carries out. Let's suppose - just as an example - I have two components, one for "dom-parsing" and the other for "sax-parsing".



Suppose now I was not on earth in the last years and I want to know if there exists a single component doing the two things.



I can select both of them, click on the shown item and I'll get xerces-j actually does both things. I might decide to use it if it fits my needs, since a single dependency is better than two, in most cases.

I also could want to know if other components rely on mine, or for instance how many components actually use one, which usually means it has great reputation. Remember the "last-gsoc-demo" component? I put "mysql-jdbc" as a dependency there. I just right click on the component, and find the components depending on the selected one. :)



CLIENT-SERVER ARCHITECTURAL VIEW (after latest modifications)



KNOWN BUGS

• troubles with SPARQL queries involving literals: searching against id and tasks is ok, versions and tags are not (yet the http://repo.url/tag/{tag} resource works fine... i had no time to investigate further befor pencils were off)


• dangling dependencies (i.e. after a delete operation) are not handled yet.

COMING SOON (random order)

• rest (not in the soa-ish meaning)


• enable license and license-style search criteria on the plugin


• associate a new perspective with the provided views


• improve repository explorer (i don't like that tree very much)


• bundled repository exploiting the eclipse embedded jetty server


• import existing metadata from maven POM or OSGi manifest (URL drag and drop from web browser?)


• address repository data access layer performance issues


• setup an update site somewhere on the globe

CHEERS
That's all for now, I really enjoyed the work, and I am confident this both- side fruitful collaboration will go on. A lot of things remain to be done on this project, and I won't let it down after Google Summer of Code stops.

I want to thank Philippe Ombredanne for mentoring me, and all the guys @ #eclipse-soc for supporting me and other students day after day. It was an invaluable experience to work with you guys.

See you online,

cheers,

Savino Sguera.

Status: Improving Eclipse Search

My greatest fear ( getting sick ) for SoC has become true this summer and it forced me to sleep / lie down doing nothing for about a week. However, I have another small addition for Find/Replace facilities.

This enhancement is Emacs style "Case Sensitive Replace". Here's the bug:

Retain Caps When Replacing - https://bugs.eclipse.org/bugs/show_bug.cgi?id=28949

It's waiting for code review.

About "Highlight All": I've received feedback from Daniel and I believe I'll complete the patch with a final blow. I should say that as a user, I find these two enhancements quite handy.

About lightweight Find/Replace: I've seen that the area below the editor is not sufficient for displaying the bar completely. It looks good under the editor when resized but the area below the editor is small in width in a vanilla Eclipse install.

To overcome this, I think that it's best to form the bar as a Trim contribution (top/bottom). This way, we'll have a pretty good amount of width and the users that want to use the bar in some other place will also be satisfied.

However we need to resolve bugs like https://bugs.eclipse.org/bugs/show_bug.cgi?id=195776 first. We need to have a bottom trim that can resize itself when contributions do not fit. I discussed the issue with Eric Moffatt and he said that we'll have TrimLayout improvements in M3 and M4. In this case, I'm thinking of creating this "lightweight Find/Replace" as a trim contribution and use the coolbar (which wraps and resizes correctly) for now. When we have Eric's improvements, we can carry it to the bottom trim as in Firefox.

Feel free to comment (especially on this last paragraph). :)

Have a nice weekend,

Parts, drugs and Fullscreen

Fullscreen is often an eyecatcher - isn't it? ;)
But I didn't wrote that to annoy you but to tell that the SWT team (especially Kevin Barnes) just committed fullscreen support for SWT shells. I'm really happy that my patches - which were part of my Google Summer of Code project - found their new home in the Eclipse CVS.
If you're one of the lucky ones working with SWT HEAD or any of the upcoming N-builds you can do something like this in order to turn your shells into real fullscreen windows:

Display display = new Display();
final Shell shell = new Shell(display, SWT.SHELL_TRIM);

Button bFullscreen = new Button(shell, SWT.PUSH);
bFullscreen.setText("switch fullscreen");
bFullscreen.addSelectionListener(new SelectionAdapter() {
public void widgetSelected(SelectionEvent e) {
shell.setFullScreen(!shell.getFullScreen());
}
}


When running this little application and clicking the button, you'll get a fullscreen shell which covers the the whole space of your active screen.
As this a really nice-to-have feature of Eclipse, I'm currently working on a new command/handler/keybinding for the workbench to allow the user to switch the IDE into a fullscreen application.

Besides this there is another major task planned for my SoC project - Detached Windows. Sometimes hated, sometimes loved. I've to say that I really love them...but only my patched version which can hold serveral ViewStacks (folders of views) in one window. It could look like the on on the right.


As I really follow the term "eat your own dogfood", I'm working on my SoC project with two screens:
Screen 1: Eclipse IDE in fullscreen mode which covers everything else on that screen
Screen 2: a detached Window which allows me to arrange my views and folder just like the workbench window itself
Interested? Here is a picture of my current desktop (caution: ~400kb, 3360px*1050px):

Groovy Plug-ins: Bundle Activation and Extension Point Processing

I faced an issue with the bundle activation last week. Namely, a bundle activator will not be started if the given bundle doesn't contain any classes. I.e. a bundle activator is provided by my plug-in and the contributor should use this class to enable scripting for the activation routines.

I reported this issue to Eclipse Equinox newsgroup, and the feedback was quite positive in sense of the solutions. Tom Watson purposed some options:

• Have ScriptExtensionProxy activate the contributing bundle using a transient flag when it creates a proxy (i.e. call Bundle.start(Bundle.START_TRANSIENT)). This would take extra work for ScriptExtensionProxy because it should probably read the Eclipse-LazyStart and/or the new Bundle-ActivationPolicy from OSGi R4.1 specification to verify the bundle is actually lazy activated.


• Make ScriptExtensionProxy abstract and force contributing groovy bundles to extend it with a real class which exists in the contributing bundle. Then when the extension is constructed it will automatically activate the bundle if it has a lazy activation policy.

The 1st option worked, but it smells like a hack. I think that the contributor should never call the Bundle#start() on its own... FWIW, this method could be available only to the framework and not to the user.
I've sticked to the 2nd option, but instead of making the ScriptExtensionProxy abstract I will leave it as it is, so that user may just declare it in plugin.xml if there's no need for activation.
ScriptBundleActivator is just a facade for calling the activation script, which I presume to be called activator.groovy for now. The start() method call just delegates the work to the start() method of the expando object in the script.

The activator would be useful if the contributing plug-in will have the need for extension point processing, as the INVITATION RULE says: whenever possible, let others contribute to your contributions.

Here's the draft Groovy code for extension point processing in the activator:

import org.eclipse.core.runtime.Platform

def expando = new groovy.util.Expando()

expando.start = { bundleContext ->
 def registry = Platform.getPluginRegistry()
 def extensionPoint =  
 registry.getExtensionPoint(
   "org.eclipse.soc.scripting.contributor.listeners")
   
 def extensions = extensionPoint.getExtensions() 
 extensions.each( { extension ->  
   extension.getConfigurationElements().each({
     element -> 
     def object = element.createExecutableExtension("class")
     //do something with the object
   }) 
 } )   
}
expando   // this is the script returning object

While shaping the code above I realized what is the real power of such scripting: NO CASTING! Truly, I didn't think about it before.
Now I have the following:

1. A proxy for declaring the extensions: read here and here


2. A bundle activator facade.

This is the basic functionality which may be used for groovy plug-in creation (with some limitations). Currently I'm preparing a demo which could demonstrate how these base classes could be used.
Next, I would like to provide some usability enhancements to the core plug-in: it would be cool to modify the script even if it is already packed and deployed.
P.S. I have also blogged about the progress in my own blog.

Status, IDE Generator (Miguel Garcia)

I've been refactoring, adding functionality, yet I had the uneasy feeling that something was missing ...

Big revamp of Chapter 1 ("Frameworks for text editors: JFace Text and Gymnast Runtime") of the project report. With that, a user of the IDE generator is also in a position to understand (and further customize) the generated IDE. Features are classified, code snippets and references to further literature given.

----------------------------------------------------------------------

The features per category are:

a) The Gymnast Runtime extension to the JFace Text framework

• background parsing and notification of changes to the CST of the document
• context outline (with nodes implementing navigation back to the text region, and property source)
• custom problem markers
• parse-tree view

b) Customizations requiring CST or AST

• hovers over text fragments
• hovers over annotations (annotations on the vertical ruler)
• hyperlinks
• mark occurrences
• select in outline
• show in
• context menu on the editor surface, for example Go to Declaration
• views, for example EMF Type Hierarchy
• content formatting
• content assist

c) Customizations directly working on the document text

• document partitioning
• show range indicator
• bracket matching (brace, parenthesis, etc.)
• AutoEdits (for example, SmartBrace: after typing { in the JDT editor, an indented newline and a closing } are added below it). Another example is AutoIndent.
• Double-click strategy (in JDT, double-clicking an opening brace selects all the text up to the matching closing brace)
• Automatic Indentation

d) Document-independent customizations

• templates, including associated Preferences page
• actions on the toolbar and menu bar, in particular retargetable actions (i.e., those actions whose UI activators are shared between different editors, such as e.g. Content Assist)
• new file wizard, so that the document does not start empty but in syntactically valid state (thus avoiding spurious parse errors the moment background parsing kicks in)
• Preferences
  

----------------------------------------------------------------------

It's incredible the amount of detail hiding in each feature ... an Eclipse-based text editor can really be a machine with lots of moving parts ...

http://www.sts.tu-harburg.de/~mi.garcia/SoC2007/draftreport.pdf

Status: Improving Eclipse Search

Last week, I decided to implement "Highlight All" feature mentioned in Bug 29692, since it's a common enhancement request. I believe I've produced a patch that solves it. However I'm planning to implement "soft matches", that is if the user requests it, highlighting case-insensitive matches with a different color, case-sensitive whole word matches with a stronger color etc. like Ctrl+H Search does.


About the "lightweight find" ( Bug 99294 ), you should read Comment #24 about my current thoughts. The only problem with having "Find" and "Replace" fields on the bar simultaneously is visibility of all labels,buttons,fields on the bar at the same time on a vanilla install. Iconizing "Find", "Replace", "Replace/Find", "Replace All" buttons would save some space (these button labels can be written as tooltips). About "Options" group presentation problem, I thought of a bar-switch behaviour or a second bar appearing on the Find/Replace bar.

Another solution would be adding a second bar to appear for Replace and Options (shared use). The scenario would be like this:

1. Press Ctrl+F. You'll see a bar under the editor.
Only the Find bar:

[x] Find: [_____] [Next] [Previous] [Options...] [Replace...]

2. Press Ctrl+F again. Another bar will appear under the Find bar. This should be Replace bar. Find bar will "reform" itself.

[x] Find: [_____] [Options...]

[x] Replace with: [_____] [Next] [Previous] [Replace] [Replace All]

3. Pressing button will switch the second bar (Replace bar) with an "Options" bar.

It's still very open to discussion so I'm planning to implement it after we settle on an idea.

I hope to tackle other usability issues next week together with lightweight Find/Replace design.

That's all. Thank you for your feedback.