When to Use Libraries: The Pragmatic Answer September 23rd, 2003

Tom Klaasen recently suggested using the IOUtil library to ease the work Java developers must do to create a String that contains the entire content of an InputStream. His suggestion looks set to spark some lively discussion.

I believe the discussion is a subtype of a more general argument, one that I don't feel deserves the time it enjoys.

First, there are times when it makes sense to store the entire output of a stream in a String. One very good example is the case where you'd like to execute fairly complex regular expressions over the entire String.

Once you've accepted that there are times when this is a necessary operation, the rest of the argument boils down to whether or not to use APIs. Not a particularly fresh debate, but one we don't seem to have been able to put behind us yet, sadly.

I think it's simple; if an API meets your needs and pays for itself, use it. If not, roll your own.

In this particular example, the conversion requires 7 lines of code: 

private String getInputStreamAsString(PlainTextInputStream urlInput)
       throws IOException
{
   // The size of the buffer depends on your application.  Either
   // size it to the typical smallest input size or generalize
   // this method to take the size as an argument.
   //
   int readSize = 512;
   byte[] buffer = new byte[readSize];

   int bytesRead;
   StringBuffer urlContent = new StringBuffer(readSize);
   while ((bytesRead = urlInput.read(buffer)) != -1) {
	urlContent.append(new String(buffer, 0, bytesRead));
   }
   String templateText = urlContent.toString();

   return templateText;
}

I'd say that an API would have to be solving more problems than just this one to pay for itself.

The code fragment above comes from a component I wrote, which has to grab and manipulate content from a remote web page. At first, I home-rolled my 7 lines of code and I believe that I made the right decision.

Then, I realized that getInputStreamAsString() was a necessary artifact of my decision to fetch web page content using the despicable java.net.URL and its nefarious henchmen.

When I decided to use Jakarta Commons httpclient instead, I bade a none-too-fond farewell to not only this little home-rolled carrot, but also a good 20 lines of ugly code surrounding the creation and interaction with the URL connection. I believe that this was also the right decision, and the change was painless because I originally encapsulated the dirty work properly.

My point is that it comes down to whether software pays for itself, and that whether or not it does so may well change through the lifecycle of the system. The important thing is to design your software in a way that allows you to change your mind later.

I think a lot of people don't get this because they have the time to fixate on tiny issues of principle. As you get older, you realize how short life is. That process sharpens your sensitivity to the importance of pay-off.

This reminds me, yet again, that I simply must write a review of Martin Fowler's book, "Refactoring: Improving the design of existing code", soon. Stay tuned.

0 comments »

J2EE Service Wrapper Pattern Candidate September 9th, 2003

Recently, I was tasked with converting a J2EE application's EJB references from remotes to locals, as part of the run-up to initial launch deployment on a single VM. The application features 24 entity beans and 50 session beans, all hand-coded without the use of code generation tools like xdoclet. The exercise made for one of the worst weeks of my working career.

It wasn't the pain of bringing unmaintained local interfaces up to speed with changes made only to the remote interfaces and beans. Nor was it the gruelling effort of tidying up imports and catch clauses for unthrown exceptions after the scripted regular expression edits on client code. It wasn't even the changes that couldn't be tested at compile time, like replacing casts. No, all of this was bearable.

The real soul sucker was knowing that I wasn't making it any easier to handle the undoing of the changes one day when the application outgrows a single VM.

This seems to be a problem that isn't addressed by a published J2EE design pattern. We have the Service Locator pattern, which encapsulates the details of acquiring a reference to a service in a distributed environment. What's missing is a Service Wrapper, that encapsulates the locality of a service.

I think this is a problem to which we should apply the OO design mantra, "Encapsulate the concept that varies." What varies between the use of local and remote references to EJB homes and EJB objects?

Were it not for these differences, client code would be able to interact with remote and local interfaces interchangeably without modification. While type-based conditional behavior could be introduced into the client, this would result in serious code duplication and degradation of business focus in the client code. So that's not the answer. But these differences can be encapsulated, and I'm suggesting that, in sufficiently large systems, they should be.

Right, enough nancying about.

What locality-related implementation specifics are we trying to isolate client code from?

This kind of wrapping is a realization of the Proxy pattern. What proxying wrappers must be introduced to guarantee isolation of client code from locality-related implementation specifics?

What's are our points of entry?

Note that two very different animals are being proxied here; EJB homes and EJB interfaces. But for each one, the same concept is being encapsulated: locality. So even though I'm proposing the introduction of a layer of indirection over two things, they form a single conceptual layer, and together comprise the Service Wrapper. Indeed, introducing one wrapper without the other doesn't solve the problem at all.

A potential point of controversy is the encapsulation of exception handling. I don't see this as a problem for two reasons:

  1. Based on my project exposure and the code excerpts I've seen from other projects and literature, it seems that most EJB applications don't respond to RemoteException with any degree of intelligence. They either allow the exception to percolate up to the client unmolested, or wrap and rethrow them in a manner that contributes nothing to the cause of failure recovery. At best, they wrap and rethrow in a way that makes it easier for developers to see the context in which failure occurred.

  2. Encapsulation of exception handling within the Service Wrapper layer can actually have failure recovery benefits. This is because the layer introduces a convenient place to put failure recovery code that would otherwise need to be scattered throughout client code. For example, the Service Wrapper layer could respond to transient network partitioning by implementing a retry algorithm, which could completely isolate clients from a significant subset of such errors, such as rebooted routing devices and remote application servers.

The class diagram below shows the introduction of one HomeWrapper and one ObjectWrapper for every EJB located through the Service Locator. This significantly increases the amount of grunt work required to write participating EJBs by hand. However, code generation tools such as xdoclet have gained tremendous support, and it wouldn't take much effort to reduce the labor to a few xdoclet tags. In fact, in the project that sparked off this line of thinking, I'm waiting for The Great XDocletization subproject to complete before applying this approach. It's just too much work to hand-code for over 70 EJBs.

In the interests of simplicity, I've only shown delegation of a single home method, create(). The important features are:

Here's a sequence diagram showing a simple interaction, in which the client requests a HomeWrapper, uses it to create a new EJB object, and accesses that object through an ObjectWrapper.

This example is simple. Wrapped finders are a little less simple, because they can return collections of EJB interfaces. In such cases, the proxying method on the HomeWrapper must return a new collection containing each EJB interface wrapped in an ObjectWrapper. Note that the code fragments in the class diagram only hint at conditional handling of locality; they don't show wrappers encapsulating variant return types.

The introduction of a Service Wrapper layer is a non-trivial exercise for large, existing applications. I expect that it's infeasible for most such applications whose EJBs are hand-coded. Mind you, if I'd thought this through properly before I did my painful remote-to-local sweep, I'd have done this as a precursor to the sweep.

Like many J2EE solutions, this is a solution to a complex problem and comes with its own complications. As always, a decision must be made as to whether the application is likely to scale to the point where this solution's additional baggage begins to pay for itself. If that point is likely to be reached, then the Service Wrapper layer is something you want in place long before you get there; it's an expensive retrofit.

I'm very interested in hearing feedback on this idea. A colleague suggested using Dynamic Proxy Classes to reduce the amount of code required, and I'll certainly be looking into that. When I've applied this solution to the project that gave birth to it, I plan to publish a follow-up, complete with doclets and sample code, a reference to the wheel I've reinvented or an admission of abject failure. I'll include any interesting feedback in that follow-up. If you send me stuff but don't want your name associated with, please mention that explicitly. I tend to assume that people want credit where it's due.

6 comments »

Friends Let Friends Write Singletons September 8th, 2003

Norman Richards recently posted a piece called Friends don't let friends write singletons, in which he expresses contempt for the Singleton pattern. At the end of what I feel is a fairly unbalanced treatment, Norman concludes with "Avoiding singletons isn't always easy."

I'd go so far as to agree that the Singleton pattern is abused. But I suggest the reason Norman has noticed that it can be hard to avoid using the pattern, is that there are situations in which it makes a lot of sense.

Fortunately, he also said: "I want to again ask why one would ever use a singleton." I like a man who expresses willingness to be proven wrong. It's a trait I'm still cultivating. So this one goes out to Norman.

I have a user activity notification component for a J2EE application. It uses MDBs to service notification requests posted to a JMS Topic. The MDBs can send notification messages by email, in which case they're delivered via an SMTP connection pool. All well and good.

But I want runtime control of the size of the connection pool. I also want to be able to query the state of the MDBs and pause and restart them. I use JMX for this. The thing is, my JMX MBean needs to communicate with two things:

  1. The SMTP connection pool. This is facilitated by making the SMTP connection pool a singleton.
  2. The MDBs. This is facilitated by introducing an MDB Observer (see the Observer pattern), with which the MDBs register themselves as Subjects. The MDBs keep the Observer up to date with their status (for monitoring), and check with the Observer before doing certain kinds of work (for control).

Both of these solutions involve the Singleton pattern, and both would be a complete ball-ache to implement without it.

So just give me a second to get a decent grip on the baby, then you can do what you like with the bath water. :-)

0 comments »

FreeBSD Decaf September 4th, 2003

I've received some horrified responses to my post on J2EE Applications on Wintendo Servers. Why on earth is a FreeBSD committer not fighting tooth and claw to put FreeBSD on a production host instead of Wintendo 2000 Swerver?

Quite simply, FreeBSD is only available in decaf; it lacks the kind of hard-hitting Java support that fuses your vertebrae, staples your eyelids to your face and performs real-time edge sharpening on your visual input.

This applies to both branches of FreeBSD (4.8-STABLE and 5.1-CURRENT), but for different reasons:

The bottom line is the same for both branches; highly threaded applications just won't scale on FreeBSD right now. J2EE applications are almost always highly-threaded for any serious production deployment. FreeBSD is fine for a Java developer; that's where I do all my development and testing work, including testing JBoss and Tomcat deployment and execution. But production? Forget about it.

And I'm not just passing on hearsay. I actually tried FreeBSD as an alternative to Wintendo for this deployment. Using the native 1.4.1p3 JDK on FreeBSD 4.8-STABLE, instantiating 2,000 concurrent Tomcat processors would result in an indefinite CPU spin. On 5.1-CURRENT, that number of processors worked, but far too slowly.

It's important to keep in mind that what I've said above has a limited shelf-life. Sooner or later, a FreeBSD release will come along that will invalidate everything I've said above. Even so, I still wouldn't have deployed on FreeBSD if we were already there. Nor on Linux, which already is there.

Why? Pragmatism, something that's seriously lacking in many Unix circles.

The technical lead on the project is most familiar with the Wintendo environment. He has a strong background in Microsoft SQL. While the SQL code used in the project is relatively RDBMS-agnostic, he recognizes that in the early days of production deployment, he's going to be fiddling a lot. He'll be fiddling with the database and he'll be fiddling with application configuration files.

Will it eventually make sense to deploy on a serious operating system? Definitely. But it doesn't make sense right now. In the throes of the product launch, most of the problems encountered are likely to be in the application itself. So what the technical lead most wants is to be freed up to focus on the application. The best way to facilitate this is to host the fledgling application on a familiar platform. As the focus of attention shifts down through the software stack, it will make more and more sense to consider moving to a serious platform, albeit unfamiliar.

Those who know me, know that I'm a vocal Microsoft detractor. Not only do I believe that the corporation produces shit software, I believe they no longer have a grip on any key area of the industry and won't even constitute a blip on the radar in a decade. But I respect the decision of the technical lead. It's a pragmatic one, and it makes.

For now...

1 comments »

Defrosting Stateless Session Beans with MBean Singletons September 3rd, 2003

Stateless session beans are often used as facades for complex or expensive operations in J2EE applications. They're relatively lightweight and benefit from pooling.

One of the problems you can run into with these babies is that facades that perform expensive operations at the beginning of their lifecycle may be very slow for the unlucky consumers whose requests give birth to them. The problem is that the J2EE specification doesn't offer you much in the way of standard ways to control the lifecycle of stateless session beans. This is a good thing, because it keeps their implementation and deployment simple. Of all the components of the EJB specification, stateless session beans are the ones that the smallest number of developers take exception to.

So then how do you get them to perform these expensive startup operations at deployment time? You don't. You can't. Not portably, and hopefully not at all. Instead, a little lateral thinking can result in a graceful solution, and the solution has become something of an idiom in the J2EE world.

Someone recently asked this question on the JBoss users' mailing list, and several people spewed the cliche: use an MBean to initiate the expensive operations in a Singleton at deploy time. Concise, elegant and totally incomprehensible to the poor bugger asking the question.

Here's a prettified version of the answer I sent him. May it serve you well.

  1. Adjust your stateless session beans to offload the expensive lookup operations to a Singleton. For example, they could invoke a method called getStuff() on the Singleton.

  2. Now factor the expensive operations out of the Singleton's getStuff() into getExpensiveStuff(). In getExpensiveStuff(), store the results in instance variables. Then adjust getExpensiveStuff() so that it uses the stored results when possible, only performing the expensive operations when necessary.

  3. In the Singleton's constructor, invoke getExpensiveStuff().

  4. Now for the MBean. An MBean is a "managed bean", part of the Java Management Extensions (JMX) specification.

    Create an MBean, called ExpensiveStuffWarmup (implementing interface ExpensiveStuffWarmupMBean), for example. Its constructor should instantiate the Singleton, indirectly invoking the Singleton's getExpensiveStuff() method. For convenience, you may want to do this with a warmStuffUp() method exported as part of the management interface, so that you can invoke it via JMX (using JBoss' jmx-console, for example).

    Chapter 2 of the O'Reilly book "Java Management Extensions" provides an excellent introduction to writing your first MBean. I downloaded chapter 2 of this book from somewhere for free a couple of months ago, but can't remember where. I've been sloppy with my notes lately. :-(

    You can view the book online if you have an O'Reilly Safari subscription. If you don't have a subscription, you may be able to view the book anyway with a 14 day trial.

  5. Lastly, create a service deployment descriptor for your MBean and arrange for it to be deployed by your application server. For JBoss, this means creating a JBoss service descriptor file called warmup-service.xml, for example, and placing it in the jboss/server/default/deploy directory.

    The service descriptor file is simple. The trick is in classloading. To keep things simple, I'd put the JAR file containing the Singleton class in jboss/server/default/lib directory, so that it's available to both the MBean and the stateless session bean, which should be deployed somewhere under jboss/server/default/deploy.

    The service descriptor might look like this: 

    <?xml version="1.0" encoding="UTF-8"?>
<server>
   <mbean code="com.example.ExpensiveStuffWarmup" name="example:service=exWarmup">
       <!-- You may need to define dependencies; I haven't never needed to before
           <depends>example:service=exDependent</depends>
       -->
   </mbean>
</server>

This may seem like a lot of work, but keep in mind that you're doing optimization work here. With J2EE, optimization is usually something that focuses outside the business logic domain and its only inside the business logic domain that J2EE is supposed to save you time. Har har, did I pull the one with bells on?

0 comments »

J2EE Applications on Wintendo Servers September 2nd, 2003

I've spent the last 6 years avoiding Wintendo administration. I've been wildly successful in this endeavor.

Then last week happened. What a nightmare. I was tasked with setting up a dual Xeon with 4GB of RAM to host a networked J2EE application (using JBoss and Tomcat) that's been in the pipeline for 2 years. Not a problem. I had to handle the installation and configuration remotely. Also not a problem. The problem was the choice of operating system: Wintendo 2000 Server.

Life's too short for this shit. Really. Thinking back to the time when people were still buying Microsoft's "lowest total cost of ownership" lie, I have to laugh.

Anyway, it was horrible. I wouldn't want my worst enemy to go through that. So in the interests of reduced global hair loss, I humbly offer my notes from the ordeal. May they serve you well.

  1. Use of the /3GB boot.ini switch radically reduces the number of PTEs available. This results in a very low limit on concurrent threads.

    If you don't need >2GB virtual address space per process, don't enable /3GB. If you can't live without it, see this article for information on tuning SystemPages to increase available PTEs.

  2. The default Wintendo 2000 TCP connection backlog is ridiculously low (20). This is a problem for when the server is configured to ramp the number of connection handlers up as load increases. See this article for information on enabling dynamic backlog handling.

  3. The default Wintendo 2000 local user socket limit is ridiculously low (5000). This is a problem when the server accesses resources (such as databases and application containers) over the network. See this article for information on increasing this limit to the defaults found on most sensible server platforms.

  4. The default Wintendo 2000 TCP TIME_WAIT delay is ridiculously high (240 seconds). This is a problem for servers establishing and releasing new connections at a high rate. See this article for information on tuning TcpTimedWaitDelay down to a more reasonable value, like 60 seconds. Note that this violates RFC 1122, although the default value of 240 seconds already violates the standard.

  5. [SPECULATION]

    Wintendo 2000 imposes an immutable limit on the number of open files / handles allowed for a process started from a CMD.EXE prompt. See the Java Service Wrapper Site for information on installing JBoss-Tomcat as an NT service using the Java Service Wrapper.

  6. [SPECULATION]

    Wintendo 2000 imposes a 48MB heap limit for processes started via the desktop (22MB for Terminal Services desktops). The solution for (5) above applies. Terminal Services, how ironic.

I don't have KB article references to support (5) and (6). I read stuff that led me to believe them, but didn't record references. However, running Jboss-Tomcat as an NT service avoided undiagnosed resource starvation problems not covered in (1) through (3).

With these changes made, a Wintendo 2000 server can be made to service a serious JBoss-Tomcat deployment configured to operate under the kind of load that Unix-like operating systems tend to handle out of the box. The host is able to service at least 2,000 concurrent connections without application server failure.

The entire experience was cosmic; I've transcended surprise. Which is fortunate, because two hours after I finished setting up the system, its motherboard died.

2 comments »