Setting up a CVS download of Eclipse so I can get to development....

CVS is available for windows at:
http://www.wincvs.org/download.html

I downloaded it today, and tried to get into the eclipse CVS site using WinCVS, but wasn't successful. It's easier just to use Eclipse directly.

Instructions for setting up CVS for eclipse, for both committers, and common slobs like me:
http://dev.eclipse.org/cvshowto.html.

Still haven't gotten all the source downloaded yet, my connection is slow for big bulk downloads. I think it's my home DSL ISP, not the eclipse servers, because it runs fast when I download from the network at work.

Dirk replied quickly to my bug comment, as he did to the original submission.

------- Additional Comment #3 From Dirk Baeumer 2005-08-29 04:24
[reply] ------- John,
something we have to keep in mind is the fact that the current "New Java File"
template is API. Clients can create a new CU using the CodeGeneration API class
which basically forwards to StubUtility. We have to make sure that current
clients will not get broken by "reusing" the current ${type_declaration}
variable. So before going down that path we need to investigate what your
approach means regarding API and backwards compatibility.
I still tend to introduce a new variable (it avoids all the API and
compatibility issues). What we can do is adding a new varaible ${type_content}
which is a sub variable like you suggested.

- - - - - - - - - - - - - - -

Dirk -
OK, I can set it up as a new variable: ${type_content}.
I am still concerned about the user experience with both the ${type_declaration} and the ${type_content} variables available. It seems it is asking for confusion along the lines of "I changed the definition in ${type_content} but the new generated class still didn't change..." because they were using ${type_declaration} and didn't notice the difference. I assume it will be easy enough to change if I get it working this way, and it looks like the API and regression tests can still work with the re-implementation. It seems to me it will be dependent on how we integrate the new changes, but since I haven't done it yet, I can't prove it one way or the other.
Got CVS set up, and I'll try to run the original tests next.

That's all for tonight. Next comes downloading the source and running the current test suite....

The eclipse committers responded to my bug right away, early in the morning after I submitted it. Here's the response:

------- Additional Comment #1 From Dirk Baeumer 2005-08-26 06:31
[reply] ------- John,
I really encourage you to investigate further in this topic and try to provide a
patch for it that can be included into Eclipse.
I favour solution one. The only porblem I see with it is that
${type_declaration} can represent a type containing more than just an empty
body. For example if you check any of the "Generate method stubs" options on the
wizard how will this go together with the type body you are providing. So you
need more than just one variable. The templete has to look more like this:
{$ype_declaration_header} {
// A comment
{$main_method}
}
So we need variable for the main method, constructor and abstract inherited
methods as well.
The "procedure" for getting code into Eclipse is as follows:
- the code should be provided as a patch against a reasonable "latest" version
of the Eclipse code (e.g. against the latest integration build). The code
can be found in the Eclipse CVS repository. The project you are looking for
is org.eclipse.jdt.ui (some basic steps how to get the source can be found at
the end of
http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/r3_2/main.html#Overall_Planning)

- you have to provide test cases as well. The tests has to go into
org.eclipse.jdt.ui.tests
- all code patches have to be provided as attachments to bug reports. We can
use this bug report for this
I am not the expert for code templates and Martin (the expert) is currently out
of office. But as a starting point you might want to have a look at the class
StubUtility which is the facade for all template based code generation.

- - - - - - - - - - - - - - - -

I read it later that day, and was excited they were so positive about the idea. Especially interesting was Dirk's idea about template variables that show where to put the generated methods inside the class. I had also spent some time considering how to place generated methods, but hadn't considered using template variables to indicate their location.

In considering how this would work, I started getting uneasy about the user experience with mutually exclusive template variables and with the template variables for the positions of methods inside the class being coupled with one declaration template variable and not another. This implies some complex and confusing user experience, so I chewed on it for a while and came up with another idea, expressed in my next comment to the bug, composed here.

- - - - - - - - - - - - - - - -

Dirk,

Thanks for your reply, and I appreciate the encouragement.

I studied the source for a few sessions and thought about your ideas of other template variables to show the location of generated methods. That makes things harder to implement, but it will definitely make a better user experience. The way I had originally thought of it (and the way I still want to implement it as a first iteration before I take on the additional complexity) would have the comments generated in the right place inside the class, but all the stub methods would just be generated at their default location, which I'm assuming (hoping) is at the top of the method. Then, for a second iteration, I'd add the template variables to indicate where the different types of generated stub methods should go, and allow them to be interspersed with the comments.

After considering your suggestions and my original ideas for a while, I became more uneasy about the user experience if we allow alternate versions of the template variables (${type_declaration} and ${type_declaration_header}) and then offer other template variables that the user has to be aware are coupled with only the ${type_declaration_header} version, and further require the user to understand they have to manually place their class brackets. It just sounds like there are a lot of failure cases and a lot of potential user confusion.

What I came up with after some headscratching is a new suggestion. Instead of offering an alternate ${type_declaration} version, why not have ${type_declaration} insert a user-definable "sub"-template like ${filecomment} and ${typecomment} currently do? That way, the template variables available for showing where the generated method stubs go in the class can be offered only in the ${type_declaration} editing context, and we no longer have to worry about the users understanding how they are coupled. Along with that, the code behind the type stub generation can put in some protective default behavior if the user forgets to add one or both brackets (default placement is right after the class declaration header for the open bracket, and at the bottom off the template for the close bracket). Finally, the default definition of the "sub"-template can be set to the current behavior, taking care of any backward-compatibility user expectations.

In addition, this leaves open the future possibility of allowing users to define different templates for the different types: interfaces, classes, annotations, enums.

Let me know if this sounds like a plan, or if anyone over there has any further suggestions, and I'll get started.

As far as process, I will get a CVS download set up. Up until now, I have been downloading the source from your download site, but that was just to do some preliminary surfing. I like to do test-driven development, so I'll try to get some bare-bones test cases written first. That way, I can work through the logistics of building and running the tests before I get very far. Or perhaps I'll just try to mimic the current behavior with the new user-definable class template variable, so the current test cases will run.

- - - - - - - - - - - - - - - -

That's all for tonight - next, as promised, I will attempt to get a CVS build up and running, and to run the current JDT tests...

Deeper into the source...

I'm a child of the Mac UI revolution of the mid 80's. My first IDE was Lightspeed Paschal, and I still think the far superior way to "know" code is to watch it do it's thing directly in a source-level debugger. I still catch myself making hidden derisive snorts when I hear excellent and talented developers extolling the virtues of printf statements for debugging. Printf statements have their place, but when you have a debugger available, they are like trying to explore your backyard through laboriously drilled holes in a neighbors fence from two blocks away.

Especially to get to know new, complex code quickly, I like to trace it a few times in a debugger and familiarize myself with the runtime flow of the design, as well as the developer's language and style.

My wife and girls went shopping today and gardening and chores were done, leaving me with some precious weekend afternoon spare time. I put Miles Davis Get Up With It on the stereo and started tracing through NewTypeWizardPage.createType() on my PDE set-up of a recent integration build of Eclipse 3.2. (While eating some fresh cantaloupe melon and a cup of Horizon blueberry yogurt. I agree with Kent Beck that "there must always be food" but further than that, development is a complete experience. Just the right mood, tunes, and food make it go so much better....)

// Even more boring part starts here....
// You may skip to where the marginally less boring part starts up again, I won't be offended...

NewTypeWizardPage.createType()
calls
StubUtility.getCompilationUnitContent()
retrieves the user defined template by calling StubUtility.getCodeTemplate() with ID of CodeTemplateContextType.NEWTYPE_ID
Despite the similar but more limited text that has already been generated and put into "content" in NewTypeWizardPage.createType(), the template text is here, a seeming duplication with some additional content and some (like the package name and imports) missing content.
calls
StubUtility.evaluateTemplate()
that immediately calls
CodeTemplateContext.evalulate(), passing the template read in above
calls
TemplateTranslator.translate(), passing the text pattern (the user entered template) from the pattern
calls
TemplateTranslator.findVariables() after "translate" strips out the dollars and braces.
this marks the positions and lengths of the identifying text of the template variables that are embedded in the translated string.
pop back up to CodeTemplateContext.evalulate(), now that all the variables have been parsed and translated, then it calls CodeTemplateContextType.resolve(), which calls TemplateVariableResolver.resolve() on each template variable to expand the text of each variable and process them into the translated string.
It builds up an ArrayList of ReplaceEdit objects, each object representing the replacement of the template variable identifying text with the actual text value of the template variable that has been processed earlier in this operation.
Then TemplateVariableResolver.resolve() creates a Document object and initializes it with the translated text pattern that was processed above (now in the "buffer" variable).
Then it creates a MultiTextEdit object, passes it the variable positions and list of ReplaceEdit objects processed above, and calls "apply" on it (which resolves to TextEdit.apply) to call a big stack of subroutines that eventually sort everything out.
TextEditProcessor.performEdits()
MultiTextEdit(TextEdit).dispatchPerformEdits(TextEditProcessor)
TextEditProcessor.executeDo()
MultiTextEdit(TextEdit).traverseDocumentUpdating(TextEditProcessor, IDocument)
ReplaceEdit(TextEdit).traverseDocumentUpdating(TextEditProcessor, IDocument)
ReplaceEdit.performDocumentUpdating(IDocument)
Document(AbstractDocument).replace(int, int, String)
Document(AbstractDocument).replace(int, int, String, long)
GapTextStore.replace(int, int, String) - this finally replaces the text.

Now pop back up to NewClassWizardPage(NewTypeWizardPage).constructCUContent(ICompilationUnit, String, String) and this newly created string is put into a new local string called "content". Went back and reviewed the relation between the calling createType "content" variable I mentioned early in this stack. It looks like that is just used to help initially create the "type stub" text that will eventually be inserted into this new "content" variable. It is not used later in createType after constructCUContent returns.
Back in constructCUContent, it creates a new ASTParser to process the newly processed source text. If all goes well, it returns the content to createType, which spreads it around a bit and then wraps up. There is a case, however, when it creates a new string buffer, and reconstructs the content from the package and type information one more time and returns that. I haven't hit that case yet, but will probably need to know about it at some point.

// Even more boring part ends here, marginally less boring part starts up again....

I know seems like a lot of useless rigamarole for those not surfing right with me. That's perfectly right - it is a lot of useless rigamarole. But it may be helpful for reference later and save me a few round trips searching in the source.

The upshot is that the action in creating a source template is all in the template variable processing. Up in createType(), the constructTypeStub() routine creates the class declaration that I'm interested in, and initializes some variables for subroutines, then tosses everything down to constructCUContent() that grinds through the already created information to sort everything into its place. This suggests the way to get my changes to work are to create a sibling to constructTypeStub() inside createType() that constructs the stub for my new template variable, which is close to the same as the original, but is missing the brackets. This way, either the original template variable or the new template variable can be called, and the program will react accordingly. There are a couple of wrinkles to investigate, however. The first is what about user behaviors that violate the implied assumptions (using the original ${type_declaration} variable implies no brackets in the user part of the template, but the new ${type_declaration_no_brackets} implies there are. What if the user doesn't comply with the brackets? What if the user puts in both template variables? And, will this work at all in contexts where methods are initially inserted into the class source (like starting from an interface, or checking some of the "create method stub" options on the wizard.) Will have to get to those in a later installment...

Copy of the text of the bug I filed into eclipse this evening.
It is logged in bugzilla as #108071.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=108071


In the "New Java Files" code template, can't add comments inside the newly created class.


Problem:
Users can't put comments in a Java class template that are to be embedded inside the class (between the class brackets), so every new class starts with the comments in the right place without requiring the user to edit each class by hand after it's generated.

e.g. I want something like this:
public class Blee {
// constants

// member variables

// constructors

// etc...
}

...but I get this:
public class Blee {

}
// constants

// member variables

// constructors

// etc...

..or if I put in my own brackets, I get this:
public class Blee {
}
{
// constants

// member variables

// constructors

// etc...
}

Steps to reproduce:
Open Window->Preferences->Java->Code Style->Code Templates->Code->New Java Files
Edit the "Pattern:" window to include something like this:

${type_declaration}
// comment

..to get the first result above, or this:

${type_declaration}
{
// comment
}

..to get the second result above.

The Pitch
This is a convenience I've been using for 10 years before I used eclipse. You start with the same file template and fill in what you need, so you always know the basic order of items in every class, and don't have to keep typing in the section headers yourself. Having all files organized the same way with section headers is a best practice that many well-respected developers recommend. Here are some references. They all present their schemes for ordering functions within a file or within a class (as the language warrants) and their recommended commenting formats.

J2EE Design and Development - Rod Johnson p.144-5 "Delimit sections of code."
Code Complete - Steve McConnell p.449 "..order the source file carefully."
C Style: Standards and Guildelines - David Straker p.125 "Sort functions by a standard scheme..."

OK, so it's a relatively small thing, but it bugs me enough I'm willing to fix it myself and contribute the change. All I need is somebody to review the change and give me some advice on logistics (e.g. do you need a CVS changelist or can I e-mail source files, what tests should I run/fix up etc...) and to help choose a preferred solution from the alternatives.

Proposed Solution Alternatives:
1.) Create a new template variable that doesn't hard-insert brackets and allows users to insert their own brackets where they choose in their template. This is my preferred solution for a couple of reasons. First, it is an addition, so it doesn't affect the current user behavior by design, and so is safe from causing regressions based on assumptions of current usage. (Of course, I understand it may cause regressions based on implementation, but I pledge to handle those.) Second, it gives the user the ultimate flexibility in controlling exactly what appears after the type declaration before the opening class brackets, what appears between the class brackets, and what appears after the class brackets. It would go something like this:

${type_declaration_no_brackets}
{
// the template class comments...
}

and could accomodate something as bizarre as this:
${type_declaration_no_brackets}
// some squirrelly thing here, perhaps a comment or future metadata tag
{
// the template class comments...
}// some other squirrelly thing, like a tool-specific embedded comment

2.) Change the behavior so the new type wizard detects if the user has inserted brackets in the template, and if so, uses these brackets instead of hard-inserting their own. This saves creating a new template variable, and seems like a sophisticated solution, but has a couple of strange corner cases. These involve things like putting brackets in a comment that is intended to go after the class, like an embedded comment that some tool would want in the code:

${type_declaration}

// xp3911{unintelligible-junk}

The code would have to be smart enough to parse comments and not include brackets in comments in its detection. Generally, this solution isn't as safe as the first because it changes the behavior of the current ${type_declaration} template variable and may violate some oddball user assumption somewhere.

3.) Change the behavior so the new type wizard considers everything below the ${type_declaration} template variable to be embedded in the class rather than below it. This was the first solution I thought of, but I like it the least because it changes the default behavior of the current ${type_declaration} template variable the most severely, and doesn't have the flexibility to accomodate the squirrelly comments the first solution can handle. Blech. Included here just for purposes of discussion.

So whaddya say guys? Will you accept one of these solutions (or a variation as we agree) if I implement it right? Isn't this what the spirit of eclipse is all about?

Citations to appeal to the better nature of the Eclipse committers.

Here are some references of developers who like to make sections in their code files like I do. They all present their schemes for ordering functions within a file or within a class (as the language warrants.)

J2EE Design and Development - Rod Johnson p.144-5 "Delimit sections of code."
Code Complete - Steve McConnell p.449 "..order the source file carefully."
C Style: Standards and Guildelines - David Straker p.125 "Sort functions by a standard scheme..."

Time to put on some Beach Boys records and surf some Eclipse code.

First, I search for *Template*.java, and get a number of hits.

Among them, I find:
org.eclipse.jdt.internal.corext.template.java.CodeTemplateContextType
which contains a definition of a constant NEWTYPE_ID that looks promising.

I want to see where the new types are created, so I right-click and look for references in the workspace for NEWTYPE_ID. It has 3 references, the first 2 of which don't lead much of anywhere, but the 3rd, "StubUtility.java" leads to another utility class called "CodeGeneration" that has a static wrapper for the StubUtility.getCompilationUnitContent method.

This method CodeGeneration.getCompilationUnitContent is called in a few places, one of which is called "NewTypeWizardPage.createType." Looks like we're getting warm. This is a pretty long function - about 150 lines, and McCabe complexity around 19. In the middle of all that, it seems to handle two major cases. First case, top-level class (or not-inner-class as the function logic goes) - second case, inner class. Then in the midst of these code blocks, when you try to find where the action really is in all the subroutines, you find calls to a method "constructTypeStub" in the same class.

It is finally here, in org.eclipse.jdt.ui.wizards.constructTypeStub, where we find the code to build the text that creates a new type. It is here where I believe I'll want to make my change to interpolate the class comments in between the class's brackets.

Now I need to make a call as to how to design the solution, and make my problem and desired solution known to the Eclipse committers. But that's for the next installment...

Surfin' Eclipse Source...

When I last left this saga, I was thinking I would try to fix the "embedded comments in new class template" problem. I started trying to look through the source to find the place in the code where the class template was inserted into the code.

The first problem was figuring out how to get the source to do the surfing. Generally, Eclipse is too big and complicated for any single person to build the whole thing from scratch. If you look through the discussion groups and search for "build from source" or some such, you'll find some soul asking the insiders how to get the last revision from CVS and build eclipse from the source. The answer refers this inquisitive soul to the general documentation front page with no further comment. This is equivalent to telling him to take a flying leap into the lake and soak his head. At least they didn't preface it with "RTF...."

The way that Eclipse supports for building and correcting itself is to allow intrepid developers to import a plug-in at a time as a source project, and make changes, build, and integrate using the PDE. This is explained in "The Java Developer's Guide to Eclipse" (D'Anjou et al, 2nd edition) on page 972. By the way, this book is turning out to be quite a helpful resource for this project.

However, if you don't know which plug-in to look in first (which I don't) you may want to get some copy of the source and surf around in it 'till you find the likely spots for your change. In this case, you can get a copy of the whole shebang of the source by means of the nightly builds. The nightly builds are not really "builds" in the sense that they don't successfully build. They are just a pull of the source from whatever happens to be at the head revision of CVS that night. Nice that they made this available to non-committers so you can surf the source without having to go through any rituals. Since we're just surfing, no need for the thing to be free of compilation errors. If you click on the "nightly build" link in the downloads page (http://www.eclipse.org/downloads/index.php), you will usually have the choice of a "source included" or a "source fetch" download, and then some versions for specific platforms. I chose the "source included" download because the source fetch was one more thing to figure out. The download is big at around 65 meg, and the servers are frustratingly slow at around 10k/sec. So it takes 2 or 3 hours a download and 1 out of 3 downloads just craps out for no good reason.

Once I find the plug-in I'm looking to change, my plan is to download the latest integration build and go the "import as source project" route, unless when I finally contact the Eclipse-o-fites, they tell me to do it a different way.

BTW, a nice summary of the different types of builds is available if you go to the download page and click on the link "For information about different kinds of builds look here."

Another BTW, one time the "source included" download was not available, and the nightly download was the same as the "integration build" download. I haven't been doing this often enough to know if it happens every integration or if it was a one-time mistake. The "source included" build reappeared the next night. There's also an option to look through the archives, but I haven't explored it yet.

Once you get a copy on your local file system, unzip the unholy mess into a directory. Then it's up to your favorite build environment. I use Eclipse, so I fired it up, created a new project, and pointed it at the source directory. After about a half hour of grinding the disk, it came up with a few hundred errors. Many of these were the "import cannot be resolved" category. These can be fixed if you need (or are obsessive) by adding the package in question to the source path. Many just look like half-implemented features.

Eclipse is awfully big for itself to run as a project, so workspace saves are lengthy. I was able to surf around successfully in the Eclipse source and started searching for the spot where templates are inserted. One nice thing to say about Eclipse (lest you think that all I do is complain) is the "References" and "Open Declaration" menu items in the drop-down menu are very handy for finding needles in haystacks.

Next - will he find this particular needle, or just a lotta hay?

Haven't been idle the past week, but haven't made big progress yet.

Read about defining new preference defaults in "The Java Developer's Guide to Eclipse" (D'Anjou et al). On page 182, it gives instructions. Haven't tried it yet, but it looks like a process only a Unix admin could love. You have to find the right plugin_customization.ini file and experiment to find out if there is a plugin default you want by changing preferences and diffing the file.

OK - so it seems a good contribution to Eclipse would be to give a way for users to edit default preferences in the UI itself, rather than having to munge around in files. Also, it seems it wouldn't hurt to have a user_customization.ini file that overlays the defaults in the active product's file so you can carry them across installs and they won't keep getting wiped. A lotta conjecture I know - need to check it out for sure. I'll try at some point to change the {$index} template variable and see if I can do it this way. At least it would solve my immediate pain while I attempt the more substantial changes required to do this fix.

That doesn't fix my problem with the class template, though. That one may be the easiest to contribute at first, because all I need to do is create a version of the same thing they have now without the brackets.

In the meantime, I tried to build the nightly build of eclipse source with eclipse. It had hundreds of errors and thousands of warnings. Hoo boy. looking at the errors, they mostly had to do with misaligned classes (have to override method x, or class Y doesn't have method z.) Eclipse as an IDE also seems to have some problems with the large number of classes in Eclipse source, and gets some errors confused. (Meaning if you click on the little error rectangle in the right side of the screen or double-click on the identifier with the red squigglies, you get an error message that doesn't match the identifier. "Method must return something" on a class name, etc...)

I wonder if I have to build the whole shebang to make a contribution - surely not. It was interesting to try. I wonder if it's because I tried the nightly and the nightly builds don't build, or if there's some magic configuration pain I have to experience before I'm worthy of a clean build.

Ahh- just looked up the definitions of the different builds on the site, and looks like nightly builds are not really builds, just whatever happens to be checked in to the head revision of source control. Not guaranteed to build. Looks like the integration or stable build is what I'd want if I wanted the thing to actually build.

OK - so onwards we go....

Gathering resources...

I made a guess the JDT project was where the action would be.
Downloads - Eclipse Project JDT Downloads - Stream Nightly Build - eclipse-sourceBuild-srcIncluded-.zip where is a big string of characters that has the date and some kind of index number in case they ever do more than one of these in a day.

Download is pretty big and takes about ten minutes to my laptop over my wireless at home.

I expanded it out into a new directory on my hard drive I called "eclipse-source."

A little digging in the source reveals that templates are part of the JDT UI project.

(Of course, I found them the first time, but couldn't remember how to get back to them the second time. Windows search with "template*.java" reveals the following folders have related content:)
plugins\org.eclipse.ui.workbench.texteditor\src\org\eclipse\ui\texteditor\templates
plugins\org.eclipse.jface.text\src\org\eclipse\jface\text\templates
plugins\org.eclipse.jface.text\src\org\eclipse\jface\text\templates\persistence
plugins\org.eclipse.text\src\org\eclipse\jface\text\templates
plugins\org.eclipse.jdt.ui\ui\org\eclipse\jdt\internal\ui\preferences
plugins\org.eclipse.jdt.ui\ui\org\eclipse\jdt\internal\ui\text\template\contentassist
plugins\org.eclipse.jdt.ui\ui\org\eclipse\jdt\internal\ui\text\template\preferences
plugins\org.eclipse.jdt.ui\core extension\org\eclipse\jdt\internal\corext\template\java
plugins\org.eclipse.pde.ui\src\org\eclipse\pde\ui\templates
plugins\org.eclipse.pde.ui\src\org\eclipse\pde\internal\ui\wizards\plugin
plugins\org.eclipse.pde.ui\src\org\eclipse\pde\internal\ui\wizards\templates
plugins\org.eclipse.team.cvs.core\src\org\eclipse\team\internal\ccvs\core\client

Daunted yet?
Turns out the file:
plugins\org.eclipse.ui.workbench.texteditor\src\org\eclipse\ui\texteditor\templates\package.html
..has a decent introduction

So it was the workbench, not the JDT.

instructions.html in the root directory has build instructions for the source.

Must register my intent to make my noble improvements...
Where to look - newsgroups? bug list? e-mail list?

Article on user editing of eclipse templates:
http://www.javalobby.org/forums/thread.jspa?threadID=15796&tstart=30