OK, so imagine you're about to buy some new J2EE application. It could be really complex and run on a full J2EE server with EJBs and JMS and XA transactions, or it could be a simple servlet engine that can run in resin or something. But whatever it comes, there comes a time when someone asks "How many users can it support per processor?" or "What is the average response time under load?"

The potential vendor can hand you a white paper like this one from SAP or this one from IBM. It will be great and list out what processors they use, amount of ram, how many users and the response time charted for various loads. If they do this then read it and size your hardware and software purchases appropriately for your new application.

However, if they, like many J2EE app vendors, say something like "Well, we have some internal numbers but nothing specific." or "We have some rough statistics, but our app is really, really fast." then run away and don't buy the product. These are code words for "We've never tested our app under load. It will probably crash under 5 users and cost you hundreds of thousands to debug and repair. Not to mention millions of huge, wasted 8 processor sun boxen."

Of course, I'm being a little facetious, but I've seen this many times. A few years ago I was looking at an app where they assured us that the app had been load tested and benchmarked. That the app ran clustered great and could support thousands of users on cheap hardware. This sounded promising so I asked to see the reports. They said they would email them right over. It's now two years later and I haven't seen any reports. Thankfully, we didn't buy the app.

Note- It looks like Oracle made a failed bid to buy MySql. Their justification was that at least when their database was getting eroded by open source, it would be their open source. This rationale also applied to the (still undenied) rumors of the JBoss buyout.