To me it’s all about removing all of the external dependencies from the code. If I can pull out all of the sections that are getting remote connections to EJBs, all of the data source lookups and queries, file reads, etc. and abstracting those out then I can test the code fairly easily because all I’ll be testing is the logic it had that surrounded all of those external calls to get data of one sort or another. So that’s what I do, I remove all of the code that does those lookups and abstract it out.

Note: Some of the feedback I’ve gotten tells me that I should make it clear that what I’m talking about below is only the refactoring part of getting your code to a unit testable state. After you perform some of the steps you might be well placed to look into either Spring or JSR-299 (for example Weld, which is their reference implementation) to do dependency injection. Or even into mock frameworks to do simpler implementations of your FooHelper for testing purposes. If you’ve got one you like a lot, let me know about it. I haven’t used any mock libraries for Java yet.

Step 1 - Use the refactoring capabilities in your IDE to extract some methods

I’m using Eclipse in this example, but I’m sure you could do much the same thing in Netbeans or any other major IDE.

Eclipse has a function on the context sensitive menu to extract a method. Just select a section of the code that goes and looks up something in a file or gets a JDBC connection and executes a query (or calls some code that does) and click the right mouse button to get the context menu, then select Refactor > Extract Method… and fill out the info in the dialog box where it asks you what to call your new method, etc. The refactoring will create an all new function which takes a set of parameters passed in and replace the code in the original spot with a call to that function. Repeat as needed on the offending section of code until you don’t have any code within it that isn’t actual logic that calls these new methods that use external resources.

Step 2 – Pull the new methods out into an external class

I don’t doubt that I could probably get Eclipse to do all of this too but at the moment I perform the next step manually.

Let’s say that the code that I was pulling methods out of was called foo(). Then I would create a new class called ProductionFooHelper at this point. I would pull all of the methods I extracted out of the class foo() was part of and put them into ProductionFooHelper(). That immediately breaks the foo() code because it no longer knows where the functions it used to call are anymore; but it leaves us with a version of foo that doesn’t require any exterior resources. That brings us to…

Step 3 – Extract an interface from the ProductionFooHelper class

Here Eclipse will do the tedious work for us again. Go to the ProductionFooHelper class and right click. Select Refactor > Extract Interface… from the context menu that pops up.

You can just select all of the methods to be in the interface and give it a name like FooHelper. Click OK and it will be created. Note that ProductionFooHelper should be marked as implementing the FooHelper interface.

Step 4 – Fix the failing calls within foo()

Go back to the class containing the foo() code and add a new parameter to it. The new parameter would be something like “FooHelper helper”. Then all the function calls within foo() that are currently showing as errors within the IDE can have “helper.” put in front of them. That will indicate that we’re calling that function on the passed in class. At this point everything in the code should resolve again except for the spot you originally called foo(). It will need to have an instance of ProductionFooHelper created and passed into it before it compiles successfully.

At this point we’ve extracted all the code which had external dependencies into another helper class and we have a production version of that class that should give us the exact same behavior we’ve always had. But if we were to produce a MockFooHelper that implemented FooHelper and produced faked results for unit tests, then we could test foo() at our hearts content knowing that we never have to do database setup or anything else unless we want to. The specific implementation of the functions in the FooHelper is up to us for testing purposes.

This may seem a convoluted way to get to testable code and maybe you’ve got much better ways to achieve this same thing. If you do, I’d love to know about it so I can improve my own coding skills. But if not, I have to say that it actually works. I’ve used it recently in a hierarchical fashion where innermost code had a helper interface extracted, then another close to it, and then yet a third. Then I wanted code that called all of them to be unit testable so I extracted a new interface that implemented the other three interfaces and added all new functions as well. The production helper had all of the methods for the innermost stuff as well as the outermost. The result is that I now have four different test points in the code where I can control everything that goes in and out and I was able to whip off a dozen new tests for code that once had none due to its complexity and the difficulty of setting something up.

Anyway, the heart of this is good, it’s in a good place. There’s no longer one big file, we can have several files and each one has its own purpose. You might even get a web.xml file that comes with some third party framework you’re using like Struts. However, then it starts to go off the rails (no pun intended):

However, because Servlet 3.0 enables you to modularize your deployment descriptors, the order in which these descriptors are processed can be important. For example, the order in which the descriptors for an application are processed affects the order in which servlets, listeners, and filters are invoked. With Servlet 3.0, you can specify the order in which deployment descriptors are processed.

Servlet 3.0 supports absolute ordering and relative ordering of deployment descriptors. Your specify absolute ordering using the <absolute-ordering> element in the web.xml file. You specify relative ordering with an <ordering> element in the web-fragment.xml file.

Uh huh. When Rails (as in Ruby on Rails) needed to order a bunch of stuff like migrations, they made a simple convention (for example, 001_filename.rb, 002_filename.rb, 003_filename.rb). When that convention didn’t work out for large groups of developers they changed and went with the date and time instead (as in, 0080402122512_one.rb). Rails pretty much always picks a convention whenever possible and then if they need to they come up with a way to override it.

There’s nothing wrong with having <ordering> and <absolute-ordering> elements at all. Give the developer an override, fine, but give them an easy way to just name their stuff and order it automatically without that. What the heck is wrong with files named 001_web.xml and 200_web.xml? Seriously? Why can there never ever be a convention that the software reacts to and uses so you don’t have to hack up some configuration file 100% of the time?

Yahoo's biggest competitor is arguably Google. Google invented an algorithm for data processing called MapReduce. They use it to process the terabytes and petabytes of data they grind through on a regular basis. They piggy back that on top of their storage system called GFS (Google File System). Because Google published papers on all this software, even though they don't make the software itself available, there was enough description for people to start developing their own versions of the Google tools.

Yahoo has now decided to both use and endorse the toolset Hadoop. Hadoop encompasses implementations of both GFS and MapReduce so arguably Yahoo is now running software that is based on ideas from their direct competitor. They aren't shy about it either, they aren't hiding it, rather they are telling the world that the software is good, they like it, and they intend to support it.

Bravo. I'm impressed.

Sun is now making all their technical sessions from Java One available online. You can get the PDF for any of them and they are gradually getting the full audio plus slides up (e.g. all the 2006 presentations and about 20-25% of the 2007 presentations are up in full audio/slides form now, the remainder of the 2007 stuff is PDF only for the moment). You have to join the Sun Developer Network (SDN) but signup for that is free.

http://developers.sun.com/learning/javaoneonline/

The presentations are pretty awesome. I'm watching the one on the Java Persistence API from 2006 and there are several others I have my eye on for later. This is an excellent educational resource and if you develop in Java for a living you would be remiss not to go through what's available there.

Some of the titles that caught my eye:

Privacy Preserving DataMining

Turning Email Upside Down: RSS/Email and IM2000

Strike Up The Brand: How to Design for Branding

Ruby And Google Maps

Ruby Sig: How To Design A Domain Specific Language

Note: I'm not endorsing any of these, I haven't had a chance to view them yet. They just looked interesting to me.

So I set out to correct it. I'm still pretty ham-handed when it comes to typing in a regular expressions and I end up having to consult cheatsheets in order to remember the syntax to do a lot of things, but I discovered online tools and websites that helped me overcome the gap in my knowledge. Here are my favorites:

• txt2re: headache relief for programmers - This lets you input a piece of text you want to match against using a regular expression and it displays different expressions you could use depending upon which parts you want to match against. It can give you a quick start on matching even if you aren't yet very regex savvy.
• RegEx: online regular expression testing - This is the other handy part of the equation. Here you can input several items of sample text and a regular expression you wish to test. Hit the test button and it will show you which test text matched, what parts were matched, etc. It's somewhat Java oriented but might be as useful for any language with fairly standard regex syntax. I've found it very helpful for iterative development of complicated regular expressions.
• Regular Expression Library - A library of already crafted regular expressions (with various levels of complexity and robustness) to validate things like email addresses and telephone numbers.
• Regular Expression Tutorial - A tutorial capable of teaching you enough to be dangerous in a fairly short time. Good info and it's not intimidating.

Oh, there's been talk about doing something for ages, but real action?!? Real dates with real pieces of software attached to the dates? Real dates that are in the near future?!? Wow! Simply WOW!

I've long argued that Sun was a company that did not get open source. Even though they had projects like NetBeans and OpenOffice, they frequently showed off a real ignorance of what we as Java users and they as Java owners stood to gain by making it all one big group effort. But when Jonathan Schwartz was put in as CEO there was a lot of talk about the fact that he did get it. Clearly the people who said that were right.

Expect Java's already huge lead on .NET to become insurmountable, this will likely be the final nail in the coffin for that particular mistake. Also expect to see releases of Java come much more frequently as we get group effort from corporations with a vested interest in Java throwing resources at it (i.e. IBM, BEA, Google, etc.).

More details are available in this article: Sun expands open-source Java plan | Tech News on ZDNet

Reaction: OH MY GOD! They bundled a database! Kill it. Kill it with fire!

They took that to its next level by releasing a beta version of a Java toolkit for Ajax work called the Google Web Toolkit. It offers a lot of the same kind of remote procedure calls and DHTML/JavaScript stuff we've seen before in nifty toolkits like DWR except that this time I see just a few things I haven't seen elsewhere. One of those things is that when you do a web interface which has multiple tabs and you use their toolkit, you can actually bookmark individual tabs and even use next and previous in the browser correctly. This is something that is broken on many websites that use Ajax today.

I haven't put it to use yet but I will be trying it out in the future and comparing it to DWR unless someone else beats me to it.

Both of the latter two perceptions, I won't even refer to them as problems, are being addressed by people like Romain Guy who are working hard at technology demonstrations that show what Swing is really capable of and how cool and sexy it can be. If you want to see some pictures of the latest demonstration of what Swing is capable of, you can check out Aerith, a Very Cool Swing Demo. It features all kinds of 2D, 3D, and other effects to look like the latest thing from Apple.

Go there, check it out and maybe it'll influence you to try something in Swing that you might not have otherwise considered.

• Eclipse Platform 3.2 M6 New and Notable
• M5 New and Notable
• M4 New and Notable
• M3 New and Notable
• M2 New and Notable
• M1 New and Notable

What we need at this time are a small army of Java developers putting their code snippets into this repository. In no time it could be the place to go for a quick routine to fix a problem.

Here are my snippets thus far:

• Ant 101
• Getting A Data Source From Tomcat
• Data Source 101
• Velocity 101
• Velocity 101 Unit Tests
• Email 101
• Email 101 Unit Tests

They will even give you a feed for when I post new snippets. Unfortunately they don't have feeds for individual tags (e.g. Java) working yet. That's a shame and it's actually one of several glitches I've noticed. But even with the bugs this is still a more believable code snippet repository than many I've seen over the years. The tags go a long way towards making it really searchable and usable. But it will only be really useful if _everybody_ enters in some of their favorite utility functions and classes. So go sign up this minute and put in just three Java code snippets! It will take you less than 15 minutes and we will all benefit.

If there aren't any more improvements or he never releases the source code or it never gets full text search then maybe I'll move my snippets to another repository in the future. But it needs to be one with syntax highlighting, an easy framework for inserting the code in the first place (this is very very easy), and tagging.

Just in case you run into the same problem, it was due to me installing a new version of Ant (1.6.5 in this case) and the ant.bat file, which is what Anthill runs under Windows XP, not passing through the error code to Anthill. This message from their mailing lists a few months ago includes a replacement ant.bat file you can use and it will fix your problem.

There are these things in Java called data sources. They are an object you can go to and ask for a database connection, it hands it off, you use it to perform a query or two, you close it. That's it. If any database pooling, testing of new connections, etc. takes place it is hidden inside the data source and you don't have to think about it. In fact, because some data source implementions do things like pooling and testing of connections, they will probably work better than that five line version of "how to create a JDBC connection" you found online and pasted into your code.

Just about any type of server or framework you might choose to use around your application typically makes it easy to create data sources, specify all the details of how they connect to a database, and all you have to do is perform a couple of calls to get a data source. In the majority of your code, you don't even worry about the details of how the data source was obtained, you just accept it as one of the parameters to your class or to your method and you use it.

If your application isn't running inside a J2EE server, or Tomcat, or Spring, or some other place where a data source object is easy to come by, you can use the Jakarta Commons DBCP library to quickly create one of your own (hopefully configured using parameters that can be easily changed).

Here's an example of what I'm talking about that takes its configuration from system parameters:

BasicDataSource dataSource = new BasicDataSource();

dataSource.setDriverClassName(System.getProperty("driverClassName"));
dataSource.setUsername(System.getProperty("username"));
dataSource.setPassword(System.getProperty("password"));
dataSource.setUrl(System.getProperty("url"));


There you go, a data source that has caching and all kinds of other features you can tweak if you need them. Once you've got it you can call dataSource.getConnection() and out pops the connection ready to use. If you are running inside of Tomcat or WebLogic or something like that then you have even less of an excuse. In Tomcat for example, they provide a way to create data sources inside the Administration interface, you can just go to a web page, log in and create them. Retrieving one of the data sources you created in your code is this simple:

// Obtain our environment naming context
Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");

// Look up our data source by the name we gave it when we created it. In this case that's "jdbc/EmployeeDB".
DataSource ds = (DataSource) envCtx.lookup("jdbc/EmployeeDB");


If you keep making everybody jump through hoops to set up your database connections when all we want to do is configure a data source, we'll just start rewriting your code. If you are writing a library to be used by other people, or an application that has to be configured by someone other than you, make it take a data source!