Binstock on Software

The Most Important Book of The Year

2010-11-18T00:27:00.000-08:00

by Jez Humble and David Farley

I have reviewed many books on this website and I have gone through numerous others as part of my work on the Jolt Awards, but it’s been a very long time since I’ve read a book as useful and likely game-changing as Continuous Delivery.

The basic premise of the book is that we need to move past continuous integration into a fuller cycle of activities that go beyond build and test. Specifically, this new orientation calls for building and testing on all platforms, creating and deploying the final deliverables for all platforms—with every check-in. The benefit of this approach is that the development organization at any given moment always has: 1) immediate feedback on deployment issues, 2) a deployable binary; 3) a completely automated process to build, test, and deploy on all platforms.

This simple concept—a kind of continuous integration on mega steroids—has profound repercussions, all of which make your process better. The first and most important is that you have to automate everything downstream from the coding. And the authors mean everything. The most common point where people hem and haw about automation is deployment. But Humble and Farley make it clear you have to “bring that pain forward,” and fix the process so it can be automated. (If you don’t have any idea how you might refine and automate deployment, think virtualization. Can you emulate your current systems on virtual machines and then progressively simplify deployment of the software to the point of automation? Good, you’re on your way.)

But the mechanics of deployment may be the least of your challenges (And here, the book’s name could be viewed as misleading: Deployment is only one aspect it covers.) You also have to build, run, and test the software on every platform you ship on. You’re not reasonably going to be able to do that if you have to change configurations and manually reset values for different platforms. The authors guide you to finding the one path that gets you across the river Jordan without spending 40 years in the desert of bit twiddling. The key is to use a single codebase and move the platform dependent stuff into configuration files. This is non-trivial, but the authors offer plenty of good advice.

Testing is another topic Humble and Farley explore in great depth. Testing in the context of continuous delivery is not just running unit tests and a regression suite. No , this is running all tests—unit, integration, UAT, and so on. How to automate them effectively occupies probably the largest chunk of the book. Even if you don’t accept the continuous delivery concept, this section is worth the price of admission. It’s mind-expanding, in ways that the hundreds of articles we’ve all read about agile testing on Digg and Reddit never touch on. You see very quickly how much more automation you could do and how to get from your miserable semi-manual existence to the smooth flow of full and continuous automation.

What impresses about the book is how the authors consistently work through hard problems. They are not daunted by them and there is no attempt to pas over them with hand waving. Hard things are examined in detail with a perspective that derives from the authors’ own extensive experience.

I have literally never read a better book on process. I believe that going forward, this book will redefine agile process and CI; and it will have as much influence as--I have to go back to 1999, here--Fowler’s book on Refactoring did on code.

Bluebeam's PDF Creation Tool Suite

2010-10-25T16:39:00.000-07:00

I use a variety of PDF tools in my editorial work. I frequently create, mark up, manipulate, and combine PDFs. In addition, I contribute to the open source Platpus typesetting project, whose major output format is PDFs. And the PDF plugin is my specific bailiwick. So, over the years, I've come to know a thing or two about PDFs, as well as the limitations of PDF tools.

The standard for PDF tools has been Adobe's Acrobat suite. But this suite is expensive, somewhat quirky, and at times works poorly with other tools. Acrobat plugins to Microsoft Office and Internet Explorer are especially unreliable, and they frequently make their host programs behave erratically. I always uninstall them.

This means I need to use other options to convert Word documents to PDF. There are several common solutions out there, none but one of them is completely satisfactory. For example, the Microsoft Office PDF plugin does not embed all fonts, nor does it give you the option to do so. It does not embed the Base14 fonts.

This is a design error (that is common). Here is its history. For many years, Adobe guaranteed that Adobe Acrobat Reader would provide 14 fonts (the so-called Base14 fonts) in all implementations. These fonts were Times Roman, Courier, and Helvetica typefaces (each in regular, bold, italic, and bold italic—so 12 fonts) plus a Symbol and a Dingbat font. The rule was you did not need to embed these fonts in PDF documents, because Acrobat Reader would supply them. This scheme never worked very well. Its first limitation was that not all Times Roman fonts looked the same, so the same document could look strikingly different on two different computers. A few years ago, Adobe quietly discontinued supporting Base14 fonts in Acrobat Reader. The result is that if you're creating a PDF for distribution, you must embed all fonts, even the old Base14 fonts, if you want it to maintain your original format and layout.

The Microsoft Office plugin does not have this option, so as a result PDFs you generate with it are not guaranteed to look correct on other systems. And, in fact, they frequently do not.

The PDF generator that come with Adobe Acrobat (not the Reader, but the paid tools) works better. It does offer an option to embed all fonts. However, in Word documents with many links, it fails to identify all links. And so rather than be clickable, the links show up as pure text.

To remedy this, I tested various Word-to-PDF tools and found none that consistently met all requirements until I ran into Bluebeam PDF Revu, a tool I had not previously heard of.

The first thing I noticed was that Bluebeam's plugins were stable and they worked correctly. The second thing I discovered was that Revu found all links in documents and by default, it embedded all fonts. So far, so good. The attention to small details in its PDFs are part of Bluebeam's DNA—it was designed as a tool for CAD users, so correctly rendering every detail of a document is a specialty.

Like the Adobe Acrobat toolbox, Revu provides editing capabilities, with better text mark-up tools than Acrobat. It also enables you to construct your own menu of tools for faster access to frequently performed operations. Form handling, digital signatures, etc. work exactly as expected. Multi-document processing can also be automated with the product. Adobe Acrobat Pro—the comparable offering from Adobe—retails at $449 list, and $350 at Amazon. The academic version of Acrobat can be found for the same price as the full Bluebeam Revu ($149) product. So, if you want the full range of options, better implemented than in Adobe's offering, and at a lower price, have a look at Bluebeam PDF Revu. (They offer a 30-day free trial.)

Keeping LOC and Tests in Balance

2010-02-04T00:52:00.001-08:00

The proliferation of metrics in software development threatens to take important quantitative measures and bury them beneath an avalanche of noisy numbers. Consequently, it's important to look for certain ratios and trends among the numbers to inform you whether a project is healthy. One tell-tale relation links LOCs and number of tests. These two values should grow in direct proportion to each other.

The included diagram presents the ratio of these two values for Platypus, the OSS project I work on.

As you can see, except for a few dips here and there, these numbers have stayed in lock step for the last 18 months. And, as you might expect, code coverage from these tests has similarly remained in fairly narrow range--right around 60%.

The most typical violation of this ratio is, as you would guess, a jump in LOCs without a corresponding rise in tests. This is something managers should watch out for. With a good dashboard, they can tell early on when these trend lines diverge. This is frequently, but not always, always indicative of a problem. (For example, it could be that a lot of code without tests was imported to the project.) Whatever the cause is, managers need to find out and respond accordingly.

(For the record, the tests counted in this diagram include unit tests and functional tests.)

The Limitations of TDD

2009-11-22T21:44:00.000-08:00

During the last 12-18 months, TDD has broken into the mainstream, it seems. And now, we're starting to see some backlash, as its limitations become better understood. Here is a sample discussion from Artima.com. Cédric Beust, who wrote the commentary, is not some unknown guy with a weird name. He wrote the TestNG unit testing framework, which is second only to JUnit in popularity. He also wrote the book, Next Generation Java Testing, which is probably the best book on pragmatic software testing that I've read in a long time. Here goes...

> That's an interesting point. Are you, in effect, saying
> that unit testing is overly emphasized, and at the expense
> of other forms of testing?

This has also been my experience, although to be honest, I see this problem more in agile/XP literature than in the real world.

This is the reason why I claim that:

- TDD encourages micro-design over macro-design
- TDD generates code churn

If you obsessively do TDD, you write tests for code that you are pretty much guaranteed to throw away. And when you do that, you will have to refactor your tests or rewrite them completely. Whether this refactoring can be done automatically or not is beside the point: you are in effect creating more work for yourself.

When I start solving a problem, I like to iterate two or three times on my code before I'm comfortable enough to write a test.

Another important point is that unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools.

You also bring up another interesting point: overtesting can lead to paralysis. I can imagine reaching a point where you don't want to modify your code because you will have too many tests to update (especially in dynamically typed languages, where you can't use tools that will automate this refactoring for you). The lesson here is to do your best so that your tests don't overlap.

--Cedric Beust

My Interview with Alexander Stepanov and Paul McJones

2009-08-04T14:02:00.000-07:00

InformIT.com has posted my interview with Alexander Stepanov (of STL fame) and his co-author Paul McJones. Their just-released book, Elements of Programming, tries to map algorithm implementations back to symbolic logic and algebraic theorems, thereby--in theory--improving their design and correctness.

In the discussion, we broach many topics that derive from this approach to programming.

Groovy Books

2009-07-25T15:40:00.001-07:00

I have been using Groovy to write functional tests for Platypus, the open-source typesetting project I work on. I am likely to make Groovy the default scripting language for Platypus in the next milestone. In the process, I've had to come up to speed on Groovy and I've been reading through and looking over the various Groovy titles on the market. Here's my take.

The Groovy bible today, without the slightest doubt, is Groovy in Action which at 650+ pages is also the most detailed book. Its principal limitation is that Groovy has undergone several revisions since it came out. Because of this, a second edition is being written. Early access to e-drafts of that edition are available here, although little as yet has been published.

If you'd like a shorter and more up-to-date introduction to Groovy, I recommend Programming Groovyby Venkat Subramaniam. At less than 300 pages, it's a quick read, provides all the needed info quickly, and covers all the highlights, with a good balance of detail.

Many people consider Grails to be the killer app for Groovy. It's a web framework that rides above Spring and Hibernate and removes much of the complexity of using those components. If you are learning Groovy to use Grails, then Beginning Groovy and Grailsis an excellent choice. It's clear, approachable, and teaches you enough Groovy to be able to follow the tutorial on Grails.

Once you get comfortable with basic Groovy, you'll quickly find yourself pining for a book of recipes that shows you how to quickly get basic tasks done using Groovy metaphors. There are two somewhat flawed recipe books on the market. The first is Groovy Recipesfrom Scott Davis, a well-regarded lecturer in the Groovy area. While calling itself a recipe book, it frequently diverges into tutorials and odd humor--both of which are obstacles when trying to find information. Some important topics are not covered at all, such as testing--which is one of the major areas where Groovy benefits Java. Database access is also not covered. In other areas, Davis' explanations seem to lack an understanding of what the user would be looking for. Nonetheless, I have successfully used some of Davis' recipes in my work. A good alternative is Groovy and Grails Recipesfrom Bashar Abdul-Jawad. This title is a true recipe book and very readable. The Groovy portion is too short, however, and an important section on file recipes (which does appear in the Davis book) is omitted. However, if you're learning Groovy to get to Grails, this is the best choice. And Abdul-Jawad does a good job understanding what readers are looking for.

Ideally, O'Reilly would publish one of its trademark comprehensive recipes book and we could all settle on that. However, when I contacted O'Reilly about upcoming Groovy titles, the company indicated it had none in the immediate pipeline.

That's pretty much it for Groovy books; although there are several others that focus exclusively on Grails. One publisher, Apress, seems to dominate that Grails market. The two titles above that cover Grails are from Apress as is the Definitive Guide to Grails, written by Graeme Rocher, who designed Grails. In the past I've been skeptical of Apress books due to wide variations in their quality, but the Groovy/Grails titles I've examined have been consistently of high quality.

As Groovy gains a wider audience, I expect more titles to emerge from all the technical book publishers.

The Fan programming language: compile to Java and .NET

2009-05-20T14:44:00.000-07:00

I have recently been playing with Fan, a programming language that reminds me a lot of Groovy, but has additional capabilities, such as actors. Its binaries run either on the JVM or .NET. Below is my recent column in SDTimes about the language.

In recent times, we are seeing an extraordinary proliferation of new languages. On one hand, thousands of domain-specific languages (DSLs) have been spawned by the advent of tools that facilitate their creation. On the other hand, we find an equal surge in full-scale, general-purpose programming languages.

The renaissance of these larger programming languages derives from several advances: 1) a renewed interest in dynamic languages and their benefits; 2) hardware that’s fast enough to run dynamic languages rapidly; and 3) the existence of two run-time environments—the JVM and the .NET CLR—that are widely used, well understood, and fast. As a result, we have an embarrassment of language choices that was inconceivable a decade ago.

In this column, I have previously highlight various interesting options among these languages: Ruby, Groovy, D, NetRexx, and a few others that elegantly address specific problems. Recently, I have been spending time with the Fan programming language, which while still early in its development cycle, is more finished and mature than most new languages at this point in their development.

Fan is a dynamic, OO language that runs on the JVM and the .NET CLR. It does this by generating intermediate code (called fcode) that is dynamically translated into Java bytecodes or a .NET DLL at startup. This step introduces a slight pause, after which programs run at full “native” speed for the given environment.

New languages arise because a developer needed to solve a problem that was not addressed well by common alternatives. The developers of Fan, a pair of brothers—Brian and Andy Frank—worked on embedded Java applications and found it difficult to sell the accompanying software to customers who were committed to Windows Mobile and .NET. So, they decided to write Fan to solve the problem and to keep it small enough that it could fit easily in a mobile device.

In the process, they removed language verbosity and added features they wanted. Their vision is remarkably balanced and complete. The language, on the verge of a freezing its 1.0 features, offers: dynamic typing and/or strong typing (à la Groovy), closures and first-class functions, extensive concurrency support (thread-safe classes with immutability specified, threads with built-in message passing, and actors), and elegant handling of various namespace issues. Low-level features include default method parameters, nullable data types, built-in field accessors, unchecked-only exceptions, and simplified numerics. The numerics handle the overflow problem that is the favorite of language puzzle writers: all integers are longs and all floats are doubles. So either type uses 64-bits and effectively does not overflow. Chars are 16-bit UTF entities.

A particularly interesting aspect of Fan is the libraries. As Brian Frank told me, “Solving the JVM/CLR portability was the easy part. The hard part was what to do with the libraries and APIs.” What the brothers did was to rethink the API sets, eliminate cruft, and use a different concept of grouping. Whereas .NET and Java both use a large number of packages that include moderate numbers of classes, Fan uses few packages that contains large numbers of classes. The result is that a developer can almost always can guess correctly which package to link to for a specific need. In addition, Fan has sensible, built-in library defaults. For example, all files I/O defaults to buffered.

The good design of a language can take it only so far. To succeed, it needs good tools, good docs, and an active community. The language tools (compiler, etc.) are all open source and written in Fan. The code is clean and surprisingly readable. As to IDE support, there is currently a plugin for JetBrains IDEA and one in the very early stages for Eclipse . The Frank brothers do all their coding in regular text editors.

The documents are very good. Probably, the best I’ve seen for any new language at this point and far better than much older “new” languages, such as D. The website is well organized and elegant; and the tutorials and “cookbook” entries clean and plentiful. It’s difficult to assess language community size in general, but more so with Fan because it does not figure on Tiobe, due I suspect to the difficulty of teasing out data for a language named Fan. For this reason and for richer Google search results, there is a move afoot to change the name of the language. Nonetheless, the community is definitely small and active. The latter aspect due to the responsiveness of the Frank brothers to users’ questions, requests, and defect reports.

Fan solves a lot of problems elegantly. If it continues growing as it has during the past year, I anticipate it will evolve into an attractive solution for some development organizations.

The biggest challenge right now is the early stage in which most IDE plugins are currently found. A second limitation, which is about to be fixed in the upcoming point release, is that libraries and binary modules are all placed by default in the same directory. The discussion on this point, found on the language's discussion boards, shows the attentive regard of the Frank brothers for their users as they kicked around various schemes, elicited comments, and posted thoughtful replies. It's one of the most spam-free, low-noise discussion groups I've been a part of in a long while. I expect good things from this language.

The Agile Rules in HP's Original Garage

2009-01-05T01:01:00.000-08:00

According to a recent HP poster, these were the rules in Bill Hewlett and Dave Packard's famous garage:

Believe you can change the world.

Work quickly, keep the tools unlocked, work whenever.

Know when to work alone and when to work together.

Share tools, ideas. Trust your colleagues.

No Politics. No bureaucracy. (These are ridiculous in a garage).

The customer defines a job well done.

Radical ideas are not bad ideas.

Invent different ways of working.

Make a contribution every day. If it doesn’t contribute, it doesn’t leave the garage.

Believe that together we can do anything.

Invent.

Curiously, it sounds like something the agile guys might have written (had they not written the manifesto). I prefer this wording because of its greater applicability and more dynamic presentation.

Bob Martin's "Clean Code" Reviewed

2008-11-13T19:23:00.000-08:00

I have gone through "Uncle Bob" Martin's new book, Clean Code,which is a lenthy presentation of rules that will help Java developers write better code. It's similar to Kent Beck's Implementation Patterns,except more code-fixated. Clean Code has some good points, but it contains several weaknesses that seem to have gone entirely by the reviewers on Amazon. So, here's the scoop.

First of all, it's well hidden, but the book is only partially written by Bob Martin. Many chapters are written by other consultants who work at Martin's company--many of whom I've never heard of. The one stand-out exception is Michael Feathers, whose chapter on error handling is one of the clearest in the book. I wish he had written more.

The main body consists primarily of explaining various coding rules that Martin calls heuristics and to which he assigns coded abbreviations for later reference. Alas, unlike patterns that have meaningful names as shortcuts, Martin chooses meaningless notations such as C2 and G26. So, "the function should do nothing but compacting[G30]" is a shortcut for the author, but a pain for the reader who has to cross-reference these references repeatedly to know what Martin is talking about.

Unlike Beck's book, there is no theoretical framework to Martin's prescriptions. The book is a series of examples from which he teases this rule and that. Because of this lack of framework there is a certain desultory aspect--the rules come in seemingly random order.

Some of them make you want to leap up and clap. For example, his rule that Javadoc should not contain HTML. How many times I've come to the same conclusion! I want to read comments in code easily. The small lift that HTML brings to Javadoc pages is not in anyway worth the difficulty it adds to the reading of comments in code. Bob Martin's one of the first persons I've encountered to say so unequivocally.

Other rules are good, but later contradicted. For example, Martin states that you should never leave commented-out code in place. [C5] As he points out, no one knows why it's commented out and so it remains in place forever. However, later on in an example of refactoring code per his own rules, Martin comments out large blocks of code without an explanation of how that squares with his earlier advice. (p.374)

Martin also uses questionable coding preferences. For example, all of his code uses indents of 2 columns. 2 columns? It makes every routine look like a solid chunk of code. It's clearly not a practice to be recommended.

A large portion of the book is an example of Martin refactoring someone else's code. He takes a long piece from an OSS project and proceeds to "improve" it. I found this section uncompelling. Perhaps because in Fowler's masterpiece Refactoring,each refactoring magically transforms the code. By comparison, Bob Martin's work seems journeyman-like. I didn't find the initial code interesting nor did I find Martin's cleaned-up version luminous. I was expecting a before-and-after scenario that would make me sit up and take notice. Instead, the exercise felt preachy, condescending at times, and ultimately not terribly convincing.

My last gripe addresses an inexcusable error: typos. There aren't many but they are frequent enough to be distracting. For example, Martin seemingly does not understand the difference between it's and its. (p. 272, p. 296, among others) And his code contains typos too. (p. 309). This carelessness erodes credibility. Books that preach quality should be flawless at the level of spelling and grammar.

Overall, I think some organizations can use several of Martin's heuristics as a means of boosting their in-house coding standards. But I doubt that careful coders will find much of value. Those developers will be better served by Beck's Implementation Patterns,which is based on principles and so communicates much more information in fewer words. Since my review of Beck's book, I must confess my admiration for it has deepened, and it's the volume I would recommend if you're looking to write cleaner code.

Banishing Return Status Codes

2008-09-28T18:07:00.001-07:00

The most enduringly popular post on this blog is Perfecting OO's Small Classes and Short Methods, which presents a short series of stringent guidelines to help an imperative-trained developer master OO.

If I were to add one item to the list, it would be: Don't use return codes to indicate the status of an action. Developers trained in languages such as C have the habit of using return codes to indicate the success or the nature of failure of the work done by a function. This approach is used because of the lack of a structured exception mechanism. But when exceptions are part of the language, the use of status codes isa poor choice. Among the key reasons are: many status codes are easily ignored; developers will expect problems to be reported via the exception mechanism; exceptions are much more descriptive. And finally, exceptions enable return codes to be used for something useful--namely returning a data item.

Astute readers will note that in Java, null is frequently used as a return value to indicate a problem (as in Collections). This practice subverts the previous points, and it too should be avoided. Returning a null presents code with many problems it should not have to face. The first is the risk of a null-pointer blow-up because the return value was accessed without being checked. This leads to the code bloat of endless null value checks. A much better solution, which avoids this problem, is to return an empty item (empty string, empty collection, etc.). This too communicates that no data item fulfilled the function's mandate, but it does not risk the null-pointer problem, and it frequently requires no special code to handle the error condition.

Hence, if your OO code is characterized by heavy reliance on return codes (many of which I am certain are not checked), consider rewriting it in favor of exceptions and use return statements solely for returning non-null data items.

A Parameter-Validation Smell and a Solution

2008-09-01T17:36:00.000-07:00

Last week, Jeff Fredrick and I did a day-long code review of Platypus. We used a pair-programming approach, with Jeff driving and I helping with the navigation. Eventually, we got into the input parser, which parses input lines into a series of tokens: text, commmands, macros, and comments. Macros can require a second parsing pass, and commands often require additional parsing of parameters.

Once you get a parser working well (that is, it passes unit and functional tests, and it handles errors robustly), you generally don't want to mess with refactoring it. Experience tells you that parsers have hideous code in them and wisdom tells you to leave it alone. However, we launched in.

A frequent cause of otiose code was my extensive parameter checking. Parameters were validated at every step as tokens passed through multiple levels of parsing logic. Likewise, the movement of the parse point was updated multiple tiems as the logic resolved itself back up the processing stack. This too had to be validated repeatedly.

Jeff came up with an elegant refactoring that I could not find in the usual sources. He created an inner class consisting of the passed variables, a few methods for validating them, and a few more methods for manipulating them.

This class was then passed to the methods in lieu of the individual parameters--thereby reducing the number of parameters to one or two. And because the class constructor verified the initialization of the fields, I need only to check whether the passed class was null, rather than validate each of the internal fields.

The effect was to reduce complexity of already complex code, enforce DRY, and place the validation of the variables inside a class that contained them--a set of small, but important improvements. And like many of the best refactorings, it seems obvious in retrospect.

So, if you find your class's methods are repeatedly validating the same parameters, try bundling them in an inner class along with their validation logic. You'll like the results.

The Handiest Java Book in Years.

2008-06-03T16:28:00.000-07:00

One of the constant challenges I have as a Java developer is keeping up with the numerous good FOSS dev tools. I no sooner start testing one tool and adapting my project to it, when a new one comes along. Being an analyst and naturally curious, this new product (or new release) represents a constant temptation. Is it better than what I am using? How much effort is required to try it out? What does it do better? On and on.

I can put a lot of those concerns to rest now. I just received a copy of Java Power Tools from O'Reilly and it's exactly what I've been looking for. It contains deep explanations of the principal FOSS dev tools in 10 major categories. These explanations are not two- or four-page summaries, but in-depth expositions that provide crucial info on the strengths and weaknesses of the product. The author, John Smart, then provides detailed tutorial on using the product. It's clear he's spent lots of time exploring the dark corners of each tool. And he makes good use of that knowledge in his comparisons and comments on the products.

If you want to spend an hour or so coming up to speed on what a product is about before installing it (and without having to work through the usually limited docs), this book will get you there faster and enable you get an overview of a whole lot of tools quickly and with the assurance you have a clear understanding. Here are the tools that are covered, followed by the number of pages for each one in parentheses:

BUILD TOOLS: Ant (55), Maven (60)
SCM: CVS (20), Subversion (78)
CI: Continuum (24p) Cruise Control (19) LuntBuild (32) Hudson (19)
IM: Openfire (12)
UNIT TESTING: JUnit (20) TestNG (25) Cobertura (17)
OTHER TESTING: StrutsTestCase (10) DbUnit (44p) JUnitPerf (10) JMeter (20) SoapUI (22) Selenium (30( Fest (9)
PROFILING: with Sun tools (16) with Eclipse (15)
DEFECT MANAGEMENT: Bugzilla (20) Trac (35)
QUALITY: Checkstyle (20) PMD (18p) FindBugs (12) Jupiter (18) Mylyn (14p)

All told, 856 pages of crisp, well-written explanations. A must-have reference for the bookshelf.

Is the popularity of unit tests waning?

2008-05-22T23:15:00.000-07:00

Before getting into my concerns about whether unit testing's popularity has peaked, let me state that I think unit testing is the most important benefit wrought by the agile revolution. I agree that you can write perfectly good programs without unit tests (we did put man on the moon in 1969, after all), but for most programs of any size, you're likely to be far better off using unit tests than not.

The problem is that only a small subset of developers understand that. And recent data points suggests that the number of programmers who use unit tests is not exactly growing quickly. I'll list some of the data points below that I've been developing for my column in SD Times.

1) Commercial products on the wane. Agitar was a company whose entire fate was tied to the popularity of unit testing. Despite very good products, a free service to auto-generate unit tests for your code, and some terrific exponents (especially Alberto Savoia and Jeff Frederick) to tell their story, the company closed a down a few weeks ago, essentially having come to the conclusion that it could never be sold at a price that could repay investors. So rather than ask for more funding, it closed down. If unit testing were gaining popularity robustly, Agitar surely would have come to a different conclusion.

2) Few OSS products. Except for the xUnit frameworks themselves, few FOSS tools for unit testing have been adopted. The innovative Jester project, which built a tool that looked for untested or poorly tested logic, essentially stopped development a long time ago because to quote the founder, Ivan Moore, in a comment to me "so few sites are into unit testing enough to care about perfecting their tests."

3) Major Java instructors aren't teaching it. Consider this interview with Cay Horstmann, co-author of the excellent Core Java books. (He asks, "If so many experienced developers don't write unit tests, what does that say?" In speculating on an answer, he implies that good developers don't need unit tests. Ugh!)

4) Unit testing books are few and far between. I am seeing about one new one a year. And as yet, not a single book on JUnit 4, which has been out for nearly three years(!).

5) Alternative unit-testing frameworks, such as the excellent TestNG, are essentially completely invisible. I was at a session on scripting this spring at SD West and in a class of 30 or so, two people had heard of TestNG (the teacher and I).

I could speculate on causes, but I have no clear culprit to point to. Certainly, unit testing needs to be evangelized more. And evangelized correctly. The folks who insist on 100% code coverage are making a useful tool unpalatable to serious programmers (as discussed here by Howard Lewis Ship, the inventor of Tapestry). But, I think the cause has to be something deeper than this. I would love to hear thoughts from readers in real-world situations where unit testing has been abandoned, cut back, or simply rejected--and why.

It would be a shame to have unit testing disappear and its current users viewed as aging, pining developers hankering for a technology the world has largely passed by. That would return programmers to the tried-and-true practice of glassy-eyed staring at a debugger for hours--something I have not missed at all.

Knuth Interview Posted

2008-04-25T15:01:00.000-07:00

My interview with Donald Knuth is now posted. It's a long piece, that has some unusually interesting points, including:

- why Knuth doesn't believe in designing code for reuse
- he's most unconvinced of multithreading and multicore on the desktop
- discussion of the tools he uses to program and write (including Ubuntu)
- etc.

A very fun read (and a fun interview to do).

Perfecting OO's Small Classes and Short Methods

2008-04-23T22:57:00.000-07:00

In The ThoughtWorks Anthology a new book from the Pragmatic Programmers, there is a fascinating essay called “Object Calisthenics” by Jeff Bay. It’s a detailed exercise for perfecting the writing of the small routines that demonstrate characterize good OO implementations. If you have developers who need to improve their ability to write OO routines, I suggest you have a look-see at this essay. I will try to summarize Bay’s approach here.

He suggests writing a 1000-line program with the constraints listed below. These constraints are intended to be excessively restrictive, so as to force developers out of the procedural groove. I guarantee if you apply this technique, their code will move markedly towards object orientation. The restrictions (which should be mercilessly enforced in this exercise) are:

1. Use only one level of indentation per method. If you need more than one level, you need to create a second method and call it from the first. This is one of the most important constraints in the exercise.

2. Don’t use the ‘else’ keyword. Test for a condition with an if-statement and exit the routine if it’s not met. This prevents if-else chaining; and every routine does just one thing. You’re getting the idea.

3. Wrap all primitives and strings. This directly addresses “primitive obsession.” If you want to use an integer, you first have to create a class (even an inner class) to identify it’s true role. So zip codes are an object not an integer, for example. This makes for far clearer and more testable code.

4. Use only one dot per line. This step prevents you from reaching deeply into other objects to get at fields or methods, and thereby conceptually breaking encapsulation.

5. Don’t abbreviate names. This constraint avoids the procedural verbosity that is created by certain forms of redundancy—if you have to type the full name of a method or variable, you’re likely to spend more time thinking about its name. And you’ll avoid having objects called Order with methods entitled shipOrder(). Instead, your code will have more calls such as Order.ship().

6. Keep entities small. This means no more than 50 lines per class and no more than 10 classes per package. The 50 lines per class constraint is crucial. Not only does it force concision and keep classes focused, but it means most classes can fit on a single screen in any editor/IDE.

7. Don’t use any classes with more than two instance variables. This is perhaps the hardest constraint. Bay’s point is that with more than two instance variables, there is almost certainly a reason to subgroup some variables into a separate class.

8. Use first-class collections. In other words, any class that contains a collection should contain no other member variables. The idea is an extension of primitive obsession. If you need a class that’s a subsumes the collection, then write it that way.

9. Don’t use setters, getters, or properties. This is a radical approach to enforcing encapsulation. It also requires implementation of dependency injection approaches and adherence to the maxim “tell, don’t ask.”

Taken together, these rules impose a restrictive encapsulation on developers and force thinking along OO lines. I assert than anyone writing a 1000-line project without violating these rules will rapidly become much better at OO. They can then, if they want, relax the restrictions somewhat. But as Bay points out, there’s no reason to do so. His team has just finished a 100,000-line project within these strictures.

Easy Does It With easyb

2008-04-07T22:20:00.000-07:00

I just got back from the CITcon conference, which is the thrice-yearly confab of agile developers who use continuous integration (the "CIT" in the conference name). This was my second time at CITcon. It's an open-space conference that is--surprise!--free, and chock-a-block full of good information. The principal reason it's so informative is that anyone committed enough to CI to go to a conference has probably spent a lot of time thinking about how to solve problems of build and test at his/her site. And this concern and reflection on these issues is amply evident in the discussions in the hallways and the informal presentations.

All the sessions I attended were thought-provoking. But probably the most interesting was a presentation by Andy Glover, the president of Stelligent, an agile consultancy. He runs a great blog in which has been touting a tool called easyb, which enables you to script unit tests so that they describe a scenario (rather than a code feature) and then test for the expected result. I've read Andy's enthusiasm for easyb, but it wasn't until I saw him demo it that I understood what the excitement was about.

The key benefits are 1) you can show a non-programmer (like the manager who is expecting the software any day now) that you have written tests that match every one of his requirements--easyb enables you to do this by writing the test in near English language; 2) you can test at a slightly higher level than the unit test: rather than test tiny features individually, you can quite easily test a succession of conditions that are chained together.

This approach is called--a little misleadingly,--behavior-driven development; which was an immediate turn off for me. I really don't want to learn another x-driven development. I just want to do what I do better. And I think easyb might just be such a tool. So, don't worry about the name, and hop over to the easyb website for a quick look-see. You'll like what you find.

Great Reference For Ruby

2008-03-31T18:50:00.000-07:00

Ruby aficionados have been working for the last few years under a serious handicapt: there was not good, up-to-date reference on their favorite language. Sure, the Pickaxe book provided some guidance, but it's a hybrid work--part tutorial, part reference. And the reference section was a summary, rather than an in-depth exposition.

Ever-dependable O'Reilly just released Ruby Programming Language, which is without a doubt the definitive Ruby reference. Not only is it co-authored by Yukihiro "Matz" Matusmoto, the inventor of Ruby, but it is superbly well edited, so that every page is full of useful information presented clearly. And at more than 400 pages, that's a lot of information. Couple this book with The Ruby Cookbook, which I reviewed on this blog, and you have probably the best 1-2 combination for learning and using Ruby.

Restarting the Platypus And the Lessons Learned

2008-02-19T21:34:00.001-08:00

As many of you know, I have spent much of my free time during the last 24 months working on an open-source project called Platypus. The project's goal is to implement a command language like TeX, which enables users to embed formatting commands directly into text and generate documents of typeset quality in PDF, Microsoft Word, and HTML. The aims of Platypus are to be much easier to use than Tex and to provide many features of interest to developers, especially for printing code and listings.

After approximately 20,000 lines of Java code and comments, I have concluded that I need to restart and re-architect the project. The more I code, the more I see that I am adding top floors to a leaning tower. Eventually I'll topple it. So by restarting Platypus, I hope to straighten out architectural shortcomings and deliver a better, more expandable product more quickly.

In the process of coming to this decision, I have been able to crystallize several key lessons, a few of which I could probably have seen foreseen.

PROJECT AND DESIGN LESSONS

1) It's extremely difficult to figure out where your architecture is deficient if you have never done the kind of project you're currently undertaking. The best you can do is layout some basic architecture, abide by good dev practices, and learn as you go. Alas, as in this case, it took 20K lines of work to recognize that the architecture was irretrievably flawed and how.

2) First, do the known hard parts that can't be avoided. In the case of Platypus, I knew from the get-go I wanted a full programming language for the user to employ (the lack of which is one of the major failings of TeX). Early on, I decided that language would be JavaScript (JS). And having decided that and knowing that Java 6 had built-in support for JS, I put the issue aside for later implementation. When I revisited implementing JS, I realized that my command syntax no longer worked well with JS and that some commands would have been better implemented in JS. Had I written code and worked with embedded JS from the beginning, I could have avoided these dissonances, and I would have experienced Platypus more from the perspective of the user.

3) If you have to write a parser, write it just once. I wrote a parser for basic and compound commands, but did not anticipate intricacies of the syntax for very complex commands (think of commands to specify a table, for example). When it came time to add these commands, I found myself undoing a lot of parser work and then trying to back-fit existing syntax in to the new grammar. Parsers are a nightmare to get right, so make sure you write them just once. This means planning all your syntax ahead of time and in great detail.

4) Learn how to present your project crisply. Everyone understands writing a debugger for a hot new language is a cool project that will attract contributors. But projects where there is no immediately identifiable built-in community require crisply articulated messages. I did not do this well.

PROGRAMMING LESSONS:

a) Design Java classes around dependency injection. This will make the classes work better together and help you test them well. I am not saying use a DI framework, which is overkill in many situations, just use the DI pattern.

b) When it comes to modularity for input and output processing, plug-ins are an excellent design model. They form a very convenient way to break projects into sub-projects, and they make it easier for contributors to work on a discrete part of the project.

c) Unit testing delivers extra value when you're writing difficult code. The ability to be deep in the parser and test for some specific semantic occurrence right away is pure gold. Unit testing can also show up design errors. At one point, I was implementing a whole slew of similar commands (relating to output of special characters). I'd made each special character its own class; so each character required: copying a class, renaming it, and changing two strings inside it. Then rinse, lather, repeat. Likewise, my unit tests were just copied, tweaked, and run. This obviously is not a great use of unit tests (what are you actually testing? your ability to tweak a test template?). This was a good clue that the design needs revisiting. And in fact, it did. In the new design, one class will handle all special characters.

OBSERVATIONS ABOUT TOOLS

1) Of all the Java IDEs I've used, I like IntelliJ IDEA best. And since I review IDEs regularly for InfoWorld, I've used lots of them including the high-priced pups. IDEA provides the best user experience, bar none. However, as the number of classes in Platypus climbed towards 200, I noticed IDEA became very slow. Clicking on a file to haul it into an edit window was a tedious process, sometimes taking 30 seconds or more (even with v. 7 of the product). Because of this, I will look elsewhere at restart. I expect to use NetBeans 6 (a hugely improved IDE, by the by) at least initially. We'll see how that works out.

2) I switched from Ant to Maven while working on Platypus. Maven is a much better solution for many build steps than Ant. See my blog post on this. However, I dislike both products. I find that I still have to waste lots of time doing simple configurations. Also, I also don't like using XML for configuring builds. I generally concur with Tapestry's Howard Ship, that Ivy plus some other tool might be a better solution. I'll explore this as I go.

3) Continuous Integration (CI) is a good concept and there are truly great tools out there. But outside of providing historical data about a project, CI's value is limited on a one-developer project. This especially true when builds are already being done on a separate machine and only code that past tests is checked into the repository. (Nonetheless, the historical data is reason enough to continue using it.)

There are surely other lessons to be learned, and as they come to me, I'll post them on this blog if they seem useful.

Words of Thanks

It would be quite wrong to end this post without pausing to deeply thank Jeff Frederick, who was exceedingly generous with his time and insights while I worked on this first phase and who hastened my realization of several important aspects that I've touched on in this post. Thank you!

Internal USB Ports: What do you think they're for?

2008-01-25T23:17:00.000-08:00

Earlier this week, I was being briefed by HP about some recently released workstations. As we were moving through the slide-deck, a small item caught my attention: one workstation claimed to have 2 USB ports on the front panel, 6 on the back, and 2 marked "internal." Why, I asked, would anyone want an internal USB port on a PC? Care to guess?

The answer is: for dongle keys. Yeah, they're still around and they use USB form factors. The internal aspect is interesting. It's designed so you can insert the dongle, lock the PC and nobody walks off with the dongle key.

I honestly would never have guessed.

Excellent Explanation of Dependency Injection (Inversion of Control)

2008-01-22T18:54:00.000-08:00

I've read lots of explanations of Dependency Injection or DI (formerly known as Inversion of Control) and the associated Hollywood Principle ("Don't call us, we'll call you."). They all tend to be unclear, either because they delve immediately into highly detailed explanations, or they tie the explanation specifically to one particular technology. Such that either the pattern is lost or its simplicity is. Here is clearest explanation I've found--slightly edited for brevity (from the very good Spring in Action, 2nd. Ed. by Craig Walls):

"Any nontrivial application is made up of two or more classes that collaborate with each other to perform some business logic. Traditionally, each object is responsible for obtaining its own references to the objects it collaborates with (its dependencies). When applying DI, the objects are given their dependencies at creation time by some external entity that coordinates each object in the system. In other words, dependencies are injected into objects."

I find that very clear.

Dependency Injection was originally called Inversion of Control (IoC) because the normal control sequence would be the object finds the objects it depends on by itself and then calls them. Here, this is reversed: The dependencies are handed to the object when it's created. This also illustrates the Hollywood Principle at work: Don't call around for your dependencies, we'll give them to you when we need you.

If you don't use DI, you're probably wondering why it's a big deal. It delivers a key advantage: loose coupling. Objects can be added and tested independently of other objects, because they don't depend on anything other than what you pass them. When using traditional dependencies, to test an object you have to create an environment where all of its dependencies exist and are reachable before you can test it. With DI, it's possible to test the object in isolation passing it mock objects for the ones you don't want or need to create. Likewise, adding a class to a project is facilitated because the class is self-contained, so this avoids the "big hairball" that large projects often evolve into.

The challenge of DI is writing an entire application using it. A few classes are no big deal, but a whole app is much more difficult. For entire applications, you frequently want a framework to manage the dependencies and the interactions between objects. DI frameworks are often driven by XML files that help specify what to pass to whom and when. Spring is a full-service Java DI framework; other lighter DI frameworks include NanoContainer and the even more lightweight PicoContainer .

Most of these frameworks have good tutorials to help beginners find their way.

Use Virtualization to Avoid Malware While WebSurfing

2008-01-09T22:32:00.000-08:00

In presentations at Infoworld's Virtualization Summits (slides here), I have repeatedly discussed how virtualization can prevent malware infections when you surf the web. The idea is to surf and do all transactions from inside a VM. Most attendees listen to this suggestion, but they seem primarily to be waiting for me to move onto the meat of my talk. I suspect they don't take the advice to heart because they feel they have various utilities on the alert for viruses and malware infections. However, as we see here, even well-known companies, such as Sears and Kmart, install key loggers and malware that route private data to third parties. Meaning, that even if you go only to sites you believe are known good, you can still be infected with malware.

By browsing from within a VM, you protect yourself against many malicious packages. In the ideal scenario, you use two VMs: One for important transactions where security is paramount (online banking, investment accounts, etc.) and another for all other browsing.

If either VM becomes infected, delete it, make a clone of the master VM, and resume browsing. Periodically, you should throw out the "just browsing" VM and bring over a clean instance, so that any undetected stealth malware is disposed of. You'll need to bring over your bookmarks file when you swap VMs or, if you prefer, you can use any of the tag services (del.icio.us and the like) to maintain your list of favorites.

I use VirtualPC from Microsoft, which can be downloaded for free. You can use it to run a Windows VM, but you need to make sure you have valid licenses for those VMs. (Actually, until April 1, you need no license at all. You can download a Windows VM with IE installed directly from Microsoft.) Using a UNIX/Linux VM is an alternative approach that provides three advantages over Windows: licenses are free, the VMs are smaller (less than the 750MB Windows needs, typically), and malware writers rarely target Linux, so your VMs stay cleaner/safer longer.

One version of Linux you can't use for this purpose, though, is Ubuntu, surprisingly. It does not install correctly on Microsoft Virtual PC. Despite a wealth of tips, I have not been able to find a way to get it to run. However, Novell SUSE works fine. And I am sure other distros do too.

Anyway, this rarely discussed use of virtualization enables me to surf with impunity and with no fear of being hijacked.

Beautiful Code vs. Readable Code

2007-12-11T23:08:00.000-08:00

For many years--decades actually--I was a big fan of beautiful code. I read almost everything by Brian Kernighan, Jon Bentley, and P. J. Plauger. This passion for elegant code was an attempt to re-create the rush I felt when I first read:

*x++ = *y++

in the C Programming Language. I'd never seen anything so beautifully succinct. It was luminous!

But as years passed, I read many clever algorithms, many impressive optimizations, many small tricks. And I got less and less charge from each of these discoveries. The reason, quite frankly, is that they almost always fell into one of two categories: some very elegant expressiveness in a new language (Ruby converts from Java can attest to this) or a technique that I'm not likely to ever use. In other words, I was chasing baubles.

In time, my esthetic sense turned to code clarity for its jollies. Today, if I can pick up a blob of complex code, read it in one pass, and accurately understand what it's doing; then I feel the rush again. I most often have this feeling when reading the code of great, non-academic developers. To be honest, when reveling in such moments, I frequently have the perception that my code is not like theirs. Even my best code doesn't quite snap together like theirs does. And I have wondered what I could do to improve my code clarity.

Kent Beck's new book, Implementation Patterns is a short handbook on code clarity. I have read much of it and already I recognize some bad habits that undermine my code's readability. Beck basically looks at typical coding issues and dispenses sage advice.

This means that some recommendations are best suited to beginners, while there are just enough of the other more subtle suggestions to keep the attention of a veteran who cares about clarity. For example, one poor habit I have developed without thinking about it too much is mixing levels of abstraction in the same method. So, for example (using Beck's example):

void process() {
input();
count++;
output();
}

Here the second statement is clearly at a different level of abstraction than the other two, which makes the code harder to read quickly. Beck proposes the following, which I agree is clearer.

void process() {
input();
tally();
output();
}

There are many other habits of mine that this book has illuminated. And in the half a dozen changes it will bring to my style, I think I have derived more benefit than in all the essays I've read on beautiful code.

Before leaving off, however, I should point out two caveats. The beginner-to-intermediate material dominates; so you'll need to skim over large parts of the text to extract the valuable nuggets. (However, this aspect makes it a great gift for junior programmers at your site.) A second point is that the book lacks for good editing. A book on code clarity should be pellucid; this one is not. (Consider the use of the word 'patterns,' which is highly misleading. It's not at all about patterns.) But these are forgivable issues. The book is a useful read if you share my appreciation of clear code.

Commercial CI server now available free

2007-12-09T15:06:00.000-08:00

JetBrains, the folks behind many well-loved dev tools (such as IntelliJ IDEA and ReSharper) have announced that TeamCity 3.0, the just-released version of their continuous-integration server, is now available for free for teams of up to 20 developers. TeamCity's stock in trade is ease of use (and particularly good integration with IntelliJ IDEA).

The Fallacy of 100% Code Coverage

2007-11-25T12:03:00.000-08:00

While I love RSS for aggregating feeds from various blogs, nothing beats having an expert combing through articles and posts, culling the best ones. Few people, if any, do that culling better for software development than Andy Glover of Stelligent. His blog posts weekly collections of interesting links to tools and agile development topics. It's one of the first RSS items I read. (Fair warning, Glover makes his selections livelier by describing them with terms from the disco era.)

A meme that has appeared recently in his links is a curious dialog about the value of 100% code coverage in unit tests. It was not until I read those posts that I realized there were people still unclear on this issue. So, let's start at the foundational level: 100% code coverage is a fallacious goal. Unit testing is designed to provide two principal benefits: 1) validate the operation of code; 2) create sensors that can detect when code operation has changed, thereby identifying unanticipated effects of code changes. There is no point in writing tests that do not fulfill one of the two goals. Consequently, a getter or setter should not be the target of a unit test:

public void setHeight( float newHeight )
{ height = newHeight; }

This code cannot go wrong (unless you believe that your language's assignment operator doesn't work consistently ;-). Likewise, there is no benefit in a test as a sensor here. The operation of this code cannot change. Hence, any time spent writing a unit test for this routine is wasted.

Developers who can't resist the allure of 100% code coverage are, it seems, seduced by one of two drivers:

1) their coverage tool gives classes with 100% code coverage a green bar (or other graphical satisfaction). Describing this phenomenon, Cedric Beust, the author of the TestNG testing framework, writes in his just released (and highly recommended) book, Next Generation Java Testing, that this design of coverage tools is "evil." (p. 149) And later, he correctly piles on with: "Even if we were stupid enough to waste the time and effort to reach 100% code coverage..." The problem with coding for graphical trinkets is explained in the next driver.

2) if developers attain 100% code coverage--even at the cost of writing meaningless tests--they can be certain they haven't forgotten to test some crucial code. This viewpoint is the real illusion. By basing proper testing on 100% code coverage, the developer has confused two issues. It's what you're testing and how (so, quality) that determines code quality, not numerical coverage targets (quantity). Capable writers of unit tests know that some routines need validation through dozens of unit tests. Because these tests repeatedly work the same code, they don't result in greater test coverage, but they do result in greater code quality. By pushing for an artificial goal of 100% a developer is incentivized against writing multiple tests for complex code, in order to have the time to write tests for getters and setters. That surely can't be right.

Great Book for Programming Fonts

2007-11-07T19:25:00.000-08:00

As I've learned from working on Platypus, programming font operations is one of the most complex and convoluted areas of software development you're likely to run into...ever. It's like driving to the river madness and drinking deeply, then walking around the desert naked for 40 days, all the while reassuring yourself that you must be making progress because you're still coding. It's awful.

The reasons are complex and numerous. First among these is that file formats are capricious things. Microsoft and Adobe have both published numerous font formats--some in response to market needs, others for competitive reasons, still others because of internal pressures. The second problem is that these formats are designed for use by font experts, not by developers. They often include cryptic parameters, tables within tables, and absolutely nothing that is clear or obvious save the copyright notice.

Third is the matter of encoding. There are numerous encodings of font characters. These too seem driven by reasons largely outside of need and formulated with no particular eye to future requirements. Try to figure out encodings for CJK fonts (Chinese, Japanese, Korean character sets), and you'll feel like walking around with your hair on fire. Even in simple encodings, difficulties arise. For example, Apple and Windows use different encodings in the basic character sets, which is why apostrophes in Mac-generated documents show up on some PCs as the euro symbol. Unicode? Foggettabout it. No font today implements close to all the characters. And those that come even halfway (of which none are free that I'm aware), they are huge multimegabyte propositions. In sum, fonts are a topic shot through and through with problems and treacherous details.

Until now, there has been no central reference that developers could turn to for help. Each new font (PostScript, TrueType, OpenType, etc.) required starting anew and learning the peculiarities from scratch . But a new 1000-page book by Yannis Haralambous, entitled Fonts & Encodings (from O'Reilly) has just appeared and it's the first real tie-line to sanity in the jungle of glyphs. It explains fonts, formats, and encodings in tremendous detail; along with elaborate discussions of tools. It is the defining book for technical users of fonts.

Before I discuss two limitations, I want to reiterate that this is a great book and nothing I say should override this view. However, it's not a developer-oriented book. Except for some SVG XML and some TeX, there is little source code. So, information on how to access font data and use it to lay out documents programmatically or just to print text is still left as a challenge to the reader (though the book gets you most of the way there). The book also discussed MetaFont in too much detail, in my view, because this format, which is now little used, is extensively described by its inventor, Donald Knuth. I'd have preferred more coverage of bitmap fonts, say, then re-presenting this info. But these two items aside, this is the book to get if you ever have to do anything with fonts. It'll give you hope; real hope.

Most Popular CI Servers (An informal survey)

2007-11-01T19:26:00.000-07:00

At CITCON Brussels last month, attendees were allowed to write up questions of any kind on a bulletin board and others could come by and post answer as they were moved to. One poll was: Which CI server do you use? The answers were, in order of popularity:

CruiseControl
Hudson
Anthill Pro, TeamCity (tied)

I don't have the exact vote counts, because the results were taken down before I could get the final tallies. But suffice it to say, that CruiseControl received the lion's share, Hudson a handlful, and Anthill Pro and TeamCity garnered 1 vote each.

This survey, of course, is not scientific. Despite the fact that it was a CI conference, the voters were self-selecting, and ThoughtWorks, which is the company behind CruiseControl, was well represented at the conference. (It is in fact a sponsor of CITCON.) So, high CruiseControl figures would be expected. (Even without this factor, though, I expect it would have placed first due to its quality implementation, wide industry adoption, and its considerable lead in plug-ins to various tools and packages.)

The Hudson numbers, however, are interesting and probably meaningful. Hudson is a comparative newcomer to CI. But it has been winning converts quickly due to its ease of use. If you have a small project or just want to test the waters of CI, Hudson might well be the server to use.

Anthill Pro is a high-end CI server that can be found in two versions: an OSS version and a paid version. It was not until this conference, though, that I discovered these are completely different products. They were built from different codebases and the OSS version was a one-time release that is not updated.

I was surprised that LuntBuild made no appearance, as not so long ago, its users were raving about its tremendous ease of use. Perhaps Hudson is stealing its thunder, or perhaps its users just weren't at CITCON. It's hard to say in a small poll.

CITCON Brussels 2007

2007-10-29T16:06:00.000-07:00

I recently returned from CITCON, the continuous-integration conference. It's held three times a year, once each in the US, Europe, and Asia. Even though, CITCON lasts only one evening plus one full day, it was surely one of the most informative developer events I have ever attended. And I have been to a lot of developer conferences/shows/seminars.

What made CITCON so productive was several unusual aspects:

1) registration is limited to 100 attendees, so it has a human-sized feel to it. Unlike some shows where there are several hundred attendees in a single session, at CITCON, by the third session, I knew several of the attendees in the room from previous sessions. By the end, I'd been in sessions with about half the attendees, it seemed.

2) CITCON uses the open conference format, in which there is no pre-existing list of presentations. Rather, everyone gets together the first night and proposes topics they'd like to know more about or ones they'd like to present. Then attendees get to mark the ones they'd like to go to. The sessions with the most votes are slotted for the next day. You are free to drift in and out of the sessions as you wish. In future posts, I'll discuss this format in greater depth. However, it creates an interesting blend: some sessions are presentation, others are discussion. Of these, the discussions were consistently the most interesting. For example, one session I attended (I was one of six attendees) discussed the problem of delaying commits until after code reviews. How do you handle the opposing pressures of quality enforcement vs. timely commits? Various attendees explained what they had done at their sites and what had not worked. Finally, Patrick Smith from Agitar, expressed a method he had used in consulting, and the session moved to analysis of the benefits of that approach. This kind of fruitful sharing is near impossible at regular tradeshows except at BoF sessions, which often still lack the give-and-take of a shared problem.

3) The conference sessions all fit into one day. This remarkably changes your motivation. You show up early, you go to every session, you stay late, and you hang out afterwards for social hour to go over notes with other attendees, especially those who have interest/challenges in common with you.

Focused, informative, and no talking heads. Very cool. I'll be back. Next instance is in Denver in April.

Continuous Integration Servers

2007-10-09T22:23:00.001-07:00

The authors of the recent book on Continuous Integration have undertaken a series of snapshots of CI servers, which will be a big help to everyone who is assessing CI.

There are many CI servers to choose from, as this table amply demonstrates. For myself, I am undertaking evaluations of three options; and after some preliminary research, I have decided on these finalists:

CruiseControl (the grand-daddy of them all and the defining CI server)
Continuum (because I work with Maven 2, this should be a good fit)
Hudson (admired for its ease of use)

I'll report on my findings, although the results will be colored somewhat by how well these tools work with my current project, whose CI needs are clear, but modest. Partly in preparation, I will shortly be hanging with the cool cats at CITCON, a CI conference being held October 19-20 in Brussels. Attendance is free but limited to 100 attendees. CITCON is the brainchild of Jeff Frederick and Paul Julius of CruiseControl fame, but it typically draws cognoscenti and users of other CI servers. The format is open, meaning that it's more of a series of BoF sessions than pure lecture. I'll report on the good stuff in future posts for those who can't make it to Belgium in time.

New Uses of Virtualization: Slides Here

2007-09-27T20:08:00.000-07:00

Earlier this week, I lectured at InfoWorld's Virtualization Summit on a topic that has interested me for a long time: uses of virtualization outside of the two principal use cases (server consolidation and developer testing of portability). Here is the slide deck from the presentation. It discusses security, training, demo's, desktop consolidation, and virtual appliances, among other uses.

From Ant to Maven

2007-09-17T23:03:00.000-07:00

I spent part of the last week migrating Platypus from Ant to Maven 2. This is a migration I've been itching to do for a while. I don't much like Ant, because I find I spend far too much time struggling with it.

Maven, by comparison, works on a "convention not configuration" model that centers on a specific build sequence and an expected file layout for your project. Understand these two, and Maven makes builds simple and very rich. For one, Maven downloads all dependencies for utilities and reports you want to run as part of your build cycle. There is no more wrestling with Ant's dependency errors. In addition, Maven's end product (beyond the build's binaries) is a website that it re-creates on each run; it loads the site with reports and data about your build. So, you and the team always know where things stand with the project.

The one complaint I read about Maven 2 is that it's hard to find the info you need to set it up and use it. This is actually not the case, if you know where to look. Unfortunately, it takes a lot of digging before you find that two excellent 300-page PDF tutorials are available at no cost. Plus a great introduction. So for those who need to know, here are the links to Maven support docs:

Introduction to Apache Maven 2 (35 pages, good overview, and Getting Started)

Better Builds with Maven 2 (300 pages. Very good guide. PDF only.)

Maven, The Definitive Guide (270+ pages, HTML and PDF. The PDF looks like advance sheets for an O'Reilly book. Also excellent, and somewhat more detailed than the previous book.)

Those resources should solve nearly any issue you encounter. In an upcoming column in SD Times, I describe in greater detail the benefits I have found in migrating from Ant to Maven. Try Maven, you'll like it!

So much for strong passwords...

2007-09-09T15:35:00.001-07:00

Perhaps you think Fgpyyih804423 is a strong password. As discussed here, it took OphCrack 160 seconds to break it. That's scary!

Ubuntu Everywhere?

2007-09-02T11:34:00.000-07:00

I have long felt that desktop Linux would become a reality only when you could go to a Linux gathering and find no more than a third of the attendees at the command line. In other words, as long as users are frequently at the command line, the OS is not ready for a big share of the desktop. Desktop users require ease of use.

Earlier this summer, I was at O'Reilly's Ubuntu Live conference in Portland, and the Ubuntu tribe were almost all using the GUI interface. This inflection point confirms for me Ubuntu's claim as the desktop Linux distro. (The conference was especially enjoyable because of the lack of zealotry. It was simply a conclave of the interested with no excess of the us-against-the-world mentality--a factor which made it a far more rewarding experience.)

Having secured its place on the desktop, Ubuntu is trying to move to the server, where competition is much more intense, and where the desktop origins could help as well as hurt. Time will tell.

However, the desktop roots did not preclude Ubuntu's use in Microwulf, the first-ever supercomputer for less than $2500 and first-ever under the $100/Gigaflop threshold.

For your friends who want to try Ubuntu, but who are not geeks, I highly recommend an approachable, not-too-techie intro: Ubuntu Linux for Non-Geeks: A Pain-Free, Project-Based, Get-Things-Done Guidebook from the ever readable No-Startch Press.

Good, approachable book on SOA

2007-08-21T17:20:00.000-07:00

SOA is becoming increasingly hot and lots of developers are wondering what they need to know to implement it without getting lost in the competing standards, the infinite implementation details, and the lack of robust tools.

To the rescue comes SOA for the Business Developer from Ben Margolis, which presents the core technologies of SOA without doing the usual deep dive to the lowest levels of detail. This refreshing approach enables you to read about the five central technologies (XML, XPath, BPEL, and the upcoming SCA and SDO) without it being a massive effort. These technologies are presented clearly (the style is remarkably readable) and each is highlighted with a few key examples consisting of working code. The purpose is not to convert you into an expert into any of these, but to give you enough familiarity that you understand how the pieces work, how they fit together, and from there how to go about writing a simple SOA application, should you want to.

The book is perfect for development managers who want to come up to speed on the SOA components and look at some code without getting dragged into minutia. It's also just right for real geeks who need talking knowledge of the same. The easy style, good examples, and compact size (300 pages) mean that you can go from 0-60 pretty quickly.

Recommended (with the hope that other volumes that provide such a gentle intro for existing developers will become more common).

Unboxing Gotcha in Java

2007-08-08T20:29:00.000-07:00

In a recent blog post, Java wiseman Norman Richards, points out that this innocent-looking line of code contains a gotcha that most IDEs (he checked Eclipse, I checked IntelliJ) don't warn you about:

int sucker = thisMethodReallyReturnsAnIntegerNotAnInt();

Due to autoboxing (well, technically, unboxing), this code could set sucker to null, which pretty much guarantees the dreaded NullPointerException.

What Richards doesn't say is that this code is so innocent looking (and the bug possibility so little recognized), that almost no one would write a unit test for this assignment. At least, that is, until it blows up the first time.

Good catch. OK, back to work: Ctl-F "Integer"...

Continous Integration Book is Out

2007-08-02T20:10:00.000-07:00

The first book dedicated solely to continuous integration has just come out. I've been poring over it and have learned several things. The great dearth of useful documentation for CI implementations made me very hopeful that this book would give me a wealth of new insights.

Unfortunately, I was a bit disappointed. The book does a good job of explaining what CI is and why you should use it; and it's the text I'd rely on to sway a manager who needed convincing. But after these explanations, the book wanders around. I understand the problem: it's hard to talk about CI without giving examples for a specific CI server--of which there are so many. So the authors chose to talk about other topics: specifically, build and test issues. This is a good book for best practices for build and test cycles; but alas these are treated as disjoint topics from CI.

What more disappointed me was the lack of information on choosing a CI server. Many (but not all) CI packages are given very cursory discussions in Appendix B, where they share space with discussions of Ant and Maven. The book definitely punted here when it should have done right by its readers and really explained the differences and offered guidelines on choosing properly.

Finally, a personal grouse point. This book follows a fad in computer books of putting an epigraph at the start of every chapter. Properly chosen epigraphs should be 1) witty 2) incisive or 3) unexpected. Most of those in this book are prosaic. Do we gain anything from this quotation from Larry Bird "First, master the fundamentals." or from Henry Ford intoning "Quality means doing it right when no one is looking."?

Overall, I feel this could have been a great book. The thirst for this information is deep and the authors are knowledgeable. However, it didn't quite come together in this edition.

Needed Code Reminders

2007-07-31T09:47:00.000-07:00

In a recent blog post, Larry O'Brien, points out that we need more varied types of tags in our code than the overworked TODO. He suggests one that enables you to mark code intended for a future feature (despite YAGNI). Which I understand, but would strongly tend to avoid. I think the XP folks have it right in hewing closely to YAGNI.

But Larry's larger theme comes up for me as well. There are some useful tags I can think of. The most urgent to me is one that you can put on suspected cruft. Something like: CRUFT. This would suggest that maintenance needs to be done to check if this code is in use anywhere, and if not then to delete it. Not just "in use" in the technical sense, but in the sense of doing anything that's still needed.

I've also been hankering for a more graded version of TODO. Such as TODO-NOW and then plain TODO. The latter would mean "todo sometime."

IntelliJ (and presumably other IDEs) enable you to do create these custom tags and have them recognized/processed by the IDE, which is a really helpful.

A Limitation in Subversion. Any ideas?

2007-07-12T19:34:00.000-07:00

Today, I was speaking with an expert at CollabNet (the Subversion folks) about a problem I have with Subversion. To my amazement, he told me Subversion can't handle my (apparently) simple problem.

When I work on Platypus, I start by checking out my code from the public Subversion repository. I put it in my local project directory, and work there. Along the way, I might write code that breaks existing tests. At some point before I get all tests working, I want to check my code into an SCM, so that I can get back to where I am right now, if something should go wrong. Best practices in SCM forbid committing code to a project directory if it breaks tests. So, the way to do what I need, it would seem, is to use a temporary repositorty to commit to until I'm finished coding. Then, when all tests pass, I can commit the code back to the original public Platypus repository.

But Subversion does not permit checking code from one directory into different repositories. The suggestion from CollabNet was to copy the code to a temporary directory from which I could commit it to a separate Subversion repository. This, of course, is a non-starter--you don't keep two live copies of the code you're working on. That makes for all sorts of errors.

Another suggestion was to create a branch in the public repository and commit the small changes to that branch, and eventually merge the branch when all tests pass. I don't like this either, because it forces me to check non-working code into a public repository--the whole thing I want to avoid.

The only solution I've come up with is to use a different SCM (Perforce) for my local repository. Check temporary code into it. Then when everything is copascetic, check the tested code into the public Subversion directory. This works, but it's hardly elegant.

How do you solve this problem or how do you avoid it to begin with? TIA

Wikipedia: The New Google?

2007-07-09T00:28:00.000-07:00

The other day over lunch, x86 optimizing expert Rich Gerber articulated a change in his on-line searching that has also been showing up in my habits recently: I often search for a term in Wikipedia before I look in Google. And in many cases, my quest ends successfully in Wikipedia without recourse to Google or additional searching.

The difference is not so much in the quality of the information (although Wikipedia is demonstrably excellent), but in the quality of the links. For example, compare Wikipedia and Google results for Lua. Do Google first. At first blush, the results look pretty good. Then, search via Wikipedia. See those links, and you'll need no further convincing. (For the moment, I won't bring in the language tutorial and lots of other useful information found on the same page.)

As Gerber points out succinctly: Which would you rather have: a machine's interpolation of relevant links or links chosen by experts?

Office 2007: Getting the Hang of It

2007-06-27T20:39:00.000-07:00

In mid-May, I decided to switch to Office 2007, fearing that I would eventually start receiving docs in the new formats and then be forced to migrate. The expected flow of docs has not materialized (except from a few contacts at Microsoft); however, I have persevered with Office 2007.

I mostly use three apps: Word (lots), Outlook (some, as I don't use it for email), and Excel (somewhat less). So, I know Word 2007 fairly well. Initially, I hated it and turned off the new silly ribbon. After heavily customizing the icon bar with the features I use most, I began finding the ribbon more useful, and now I leave it on all the time. In addition, my fingers are finding their way around the commands quickly. As time passes, my appreciation of Word 2007 deepens. There are many neat features in the user experience that are difficult to describe without doing a screencast. Suffice it to say that the new interface does eventually make you more productive and your documents more elegant. (I don't use the collaboration features very much, I should note.)

I'm going to stay with Word 2007, which I didn't expect--two weeks ago I was still cursing myself for migrating. Now that I am more productive and enjoying Word 2007, I can honestly ask myself: Would I recommend the upgrade to someone with similar needs? Probably not. The new features are certainly nice to have, but they're not compelling enough be the sole reason for migrating. Meaning that if for some reason, I were forced to switch back to Word 2003, I would manage fine-- but I would have to adjust to the absence of convenience features.

I am one of the little green men

2007-06-21T19:47:00.000-07:00

In addition to my regular work, I have begun writing a monthly column for GreenerComputing.com, a site that specializes in discussing news and issues relating to environmental matters in computing.

The principal green issues in IT generally fall into three areas:

power consumption
regulatory compliance
hardware disposal

As anyone who follows the news these days already knows, there's lots of buzz about green, so this is a fun area to working in and a good complement to my traditional areas of professional focus.

Unlike the mainstream green topics (consumer, building design, etc.), green IT makes sense only if the bottom-line aspects also make sense. So, green IT operates within narrow constraints. And because of this, it lacks the moralizing dimension and is instead purely pragmatic. I think I'll like this aspect.

Milestone 1 of Platypus shipped today

2007-06-11T13:50:00.001-07:00

The open-source project I've been working on for months shipped its first milestone today. The project is called Platypus (for page layout and typesetting system) and it enables you to generate PDF docs (eventually HTML and RTF as well) from text files in which you embed formatting commands. It's reminiscent of TeX but updates many features from that system, adds ease of use (especially!), and eventually will add numerous report-friendly and developer-oriented features (such as language-sensitive code listings).

The current milestone is a small subset intended for early adopters who are interested enough to send feedback. It can do the following:

basic Type 1 fonts
bold, italics, underline, strikethrough
foreign characters (enough for French, German, Spanish)
left, center, right alignment and justified text
customizable page size
customizable margins
indented/unindented paragraphs
page-number footer
debugging features

Milestone 2 should ship in late Q4 2007.

Documentation, examples, source code, intended schedule, and other resources are available at the project website.

Great Nerd Stim!

2007-06-06T17:53:00.000-07:00

Writing a parser represents a kind of trophy achievement in programming, in my opinion. For years, it meant using very quirky tools such as yacc, and (later) bison. In the last few years, however, a new generation of parser generators has emerged that have eased the task considerably. Among those are CUP and JavaCC.

No tool, however, has generated more excitement than ANTLR, which is celebrated by its users for its ease of use and its ability to generate parsers in several languages. The ANTLR site has plenty of info on how to use it, but there has long been an unfilled need for a book you can bring with you to read on a long flight (and imagine the cool languages you could write, which is the real the nerd-stim).

This month, however, the pragmatic programmers filled that gap with a new book, The Definitive ANTLR Reference: Building Domain-Specific Languages . It a very readable intro to writing parsers in general, and specifically to writing ANTLR-usable language specifications. Normally, such books are long, dry, pedantic ordeals that force you to write many small tests versions before you understand enough theory to start the actual work you want to do. This book takes away much of that drudgery and makes the topic truly approachable. Actually, it sort of lulls you into the false belief that you could write a new scripting language fairly easily. Writing and implementing language specs is still not easy, but with this book and ANTLR software, it's easier than it's ever been.

Comments--How many should you have?

2007-05-28T14:19:00.000-07:00

While there is considerable conversation about how many unit tests to write (I have two recent posts on the topic--here and here--based on conversations with vendors), few people have much to say re how many comments there should be. Once the usual themes (more comments, keep comments up-to-date, avoid pointless comments) have been stated, the conversation ends. Everyone understands what relevant and up-to-date comments mean, but few will hazard a guess as to how many of them necessary.

Interestingly, the comment ratio is a key factor in one of the most interesting metrics, the maintainability index (MI). It is also a ratio that is rewarded by Ohloh.net, the emerging site for tracking details of open-source projects. Ohloh gives projects a kudo for a high percentage of comments. The question is how high is high enough to earn the kudo. According to Ohloh, the average open source project runs around 35% comments. Projects in the top third overall get the kudo. I don't know the cut-off for this top third, but I do know Apache's FOP with a 46% ratio of comments definitely qualifies.

Comment count is a metric that is particularly easy to spoof. You could do what some OSS projects do and list all the license terms at the top of each file. I've always disliked scrolling through this pointless legalese. In my file headers, I simply point to the URL of the license, which is sufficient. But all the license boilerplate inflates comment counts amazingly. So does commenting out code and leaving it in the codebase (gag!) or writing Javadoc for simple getters and setters (also gag).

But where legitimate comments are concerned, 35% is probably a good working number to shoot for. I find many OSS projects at this ratio have quite readable code. I'll be working to bring my own codebase up to this level.

Groovy Gaining Traction

2007-05-18T13:03:00.000-07:00

Java developers suddenly have a wealth of choices when it comes to dynamic languages that run on the JVM. There's JavaFX, which Sun announced at JavaOne this year, and JRuby, which Sun expects to complete sometime this year, and then, of course, there's my favorite: Groovy. Groovy makes writing Java programs far easier. It essentially takes Java and removes the syntactical cruft, leaving a neat language that makes you terrifically productive.

Because Groovy took a long time getting out of the gate, it's taken some licks in the press. However, it's clear that Java developers are catching on to its benefits. The JavaOne bookstore published its daily top-10 sales during the show. The picture on this post, shows the Day 2 list with two Groovy titles in the top 10 (at places 5 and 8). Overall, the Groovy bible, Groovy in Action, came in at number 5 for the show. Interest is definitely growing.

If you haven't tried Groovy yourself, it's definitely worth a look. Here are a couple of good overviews:

The Groovy Getting Started Guide
Andrew Glover's Overview and Tutorial on DeveloperWork

Unit Testing Private Variables and Functions

2007-05-16T22:46:00.000-07:00

How do you write unit tests to exercise private functions and check on private variables? For my projects, I have relied on a technique of adding special testing-only methods to my classes. These methods all have names that begin with FTO_ (for testing only). My regular code may not call these functions. Eventually, I'll write a rule that code-checkers can enforce to make sure that these violations of data hiding don't accidentally appear in non-test code.

However, for a long time I've wanted to know if there is a better way to do this. So, I did what most good programmers do--I asked someone who knows testing better than I do. Which meant talking to the ever-kind Jeff Frederick, who is the main committer of the popular CI server Cruise Control (and the head of product development at Agitar).

Jeff contended that the problem is really one of code design. If all methods are short and specific, then it should be possible to test a private variable by probing the method that uses it. Or said another way: if you can't get at the variable to test it, chances are it's buried in too much code. (Extract Method, I have long believed, is the most important refactoring.)

Likewise private methods. Make 'em small, have them do only one thing, and call them from accessible methods.

I've spent a week noodling around with this sound advice. It appeals to me because almost invariably when I refactor code to make it more testable, I find that I've improved it. So far, Jeff is mostly right. I can eliminate most situations by cleaning up code. However, there are a few routines that look intractable. While I work at find a better way to refactor them (a constant quest of mine, actually), I am curious to know how you solve this problem.

Reusing IDE and SATA Drives: A Solution

2007-05-10T11:42:00.000-07:00

Because I review lots and lots of tools, I find myself going through PCs pretty quickly. It's not fair to gauge the performance of a product on old hardware, so each year I buy new PCs. Over the course of years, I've accumulated lots of IDE drives from the PCs I've discarded. I rarely ever use them, but every once in a while I would like to know what's on them and whether I can reuse one of them. Unfortunately, this is a time consuming task, especially hooking up the drive to a PC that will access it.

I recently came across an elegant solution to this problem: the USB 2.0 Universal Drive Adapter from newertech. This device comes with a power supply for the HDD and a separate cable that plugs into an ATA-IDE drive, a notebook IDE drive, or a SATA drive. The other end of the cable is a USB plug. So, you attach the cord and the power the drive, plug the USB end of the cable into your PC--and the IDE drive magically pops up as a USB drive on your system, with full read and write capabilities.

I have cleaned up a bunch of IDE drives during the last week using this adapter. In the process, I've discovered it has some limitations. It did not work well on older drives. Some would not power up (but they did start up when I swapped them into a PC), and others did not handle read/writes well (Windows generated write errors), although it's hard to know if the errors come from the drive or the adapter. But for most drives from the last few years, the product worked without a hitch. Neat solution and, at $24.95 retail, a no-brainer purchase.

Further note: I increasingly use virtualization for my testbed infrastructure. When I'm done with a review, I archive the VMs. This keeps my disks mostly pristine and reusable, so I am not discarding disks with old PCs nearly as much as before. The fact that drives today are > 500GB also helps. ;-)

How many unit tests per method? A rule of thumb

2007-04-30T22:21:00.000-07:00

The other day, I met with Burke Cox who heads up Stelligent, a company that specializes in helping sites set up their code-quality infrastructure (build systems, test frameworks, code coverage analysis, continuous integration--the whole works, all on a fixed-price basis). One thing Stelligent does before leaving is to impart some of the best practices they've developed over the years.

A best practice Stelligent uses for determining the number of unit tests to write for a given method struck me as completely original. The number of tests is based on the method's cyclomatic complexity (aka McCabe complexity). This complexity metric starts at 1 and adds 1 for every path the code can take. Many tools today generate cyclomatic complexity counts for methods. Stelligent's rule of thumb is:

complexity 1-3: no test (likely the method is a getter/setter)
complexity 3-10: 1 test
complexity 11+: number of tests = complexity / 2

I like this guide, but would change one aspect. I think a cyclomatic complexity of 10 should have more than 1 test. I'd be more inclined to go with: complexity 3-10: # of tests = complexity / 3.

Note: you have to be careful with cyclomatic complexity. I recently wrote a switch statement that had 40 cases. Technically, that's a complexity measure of 40. Obviously, it's pointless to write lots of unit tests for 40 case statements that differ trivially. But, when the complexity number derives directly from the logic of the method, I think Stelligent's rule of thumb (with my modification) is an excellent place to start.

Effectiveness of Pair-wise Tests

2007-04-23T14:53:00.000-07:00

The more I use unit testing, the more I wander into areas that are more typically the province of true testers (rather than of developers). One area I frequently visit is the problem of combinatorial testing, which is how to test code where there is a large number of possible values it must handle. Let's say, I have a function with four parameters that are all boolean. There are, therefore, 16 possible combos. My temptation is to write 16 unit tests. But the concept of pairwise testing argues against this test-every-permutation approach. It is based on the belief that most bugs occur between the interaction of pairs of values (rather than a specific configuration of three, or four of them). So, pair-wise experts look at the 16 possible values my switches can have and choose the minimum number in which all pairs of values have been exercised. It turns out there are 5 tests that will exercise every pair of switch combinations.

The question I've wondered about is if I write those five unit tests, rather than the more ambitious 16 tests, what have I given up? The answer is: not much. At the recent Software Test & Performance Conference, I attended a session by BJ Rollison who heads up TestingMentor.com when he's not drilling Microsoft's testers in the latest techniques. He provided some interesting figures from Microsoft's analysis of pair-wise testing.

For attrib.exe (which takes a path + 6 optional args), he did minimal testing, pairwise, and comprehensive testing with the following results:

Minimal: 9 tests, 74% code coverage, 358 code blocks covered.
Pairwise: 13 tests, 77% code coverage, 370 code blocks covered
Maximal: 972 tests, 77% code coverage, 370 code blocks covered

A similar test exploration with findstr.exe (which takes a string + 19 optional args) found that pairwise testing via 136 tests covered 74% of the app, while maximal coverage consisting of 3,533 tests covered 76% of the app.

These numbers make sense. Surely, if you test a key subset of pairs possibilities, testing additional combinations is not likely to exercise completely different routines, so code coverage should not increase much for the tests that exceed pair-wise recommendations. What surprised me was that pairwise got such high-numbers to begin with. 70+ % is pretty decent coverage.

From now on, pair-wise testing will be part of my unit-testing design toolbox. For a list of tools that can find the pairs to test, see here. Rollison highly recommended Microsoft's free PICT tool (see previous link), which also provides a means to specify special relationships between the various factors in the combinations.

Update to my review of Java IDEs

2007-04-16T10:21:00.000-07:00

My review of three leading enterprise Java IDEs appeared in later March in InfoWorld. I've received some nice comments on the piece and a few corrections. Most of the corrections come from Sun in my coverage of NetBeans 5.5. Here are the principal items:

I complained that NetBeans does not use anti-aliased fonts. I overlooked a switch that can turn on these fonts in Java 5. On Java 6, they're on by default, if your system is set with font smoothing on. (It's hard to figure why NetBeans does not default to these fonts on Java 5, as do Eclipse, IntelliJ, and most of the other IDEs.)
I recommended that potential users look at NetBeans 6.0 beta, because it adds many of the features that I complain are missing. Sun gently points out that the current version of 6.0 is not quite in beta yet, but should be in beta within the next few months. For the latest version and roadmap, go to netbeans.org
After extensive discussions, Sun convinced me that they have greater support for Java 6 than I originally gave them credit for. I originally wrote they provided 'minimal' coverage. In retrospect, I would say 'good' support for Java 6.
Finally, Sun was kind enough to point out that I give them too much credit in saying that they have deployment support for JavaDB. In fact, they provide only start/stop control from within NetBeans.

If there are further corrections, I'll update this post.

InfoWorld Moves to Online-Only

2007-03-27T03:41:00.000-07:00

InfoWorld magazine announced today that it would be abandoning the print publication and going to an entirely online format. This move continues a trend that has been in place for many years in technical publications. I understand the economics of the move and think that on that basis, it's the right move. However, I confess some sadness at this transition. I like print media. I am currently overseas. And it was a great pleasure to pack magazines with my carry-on luggage and read through pages of material on the plane. SD Times and Doctor Dobb's are the only developer mags I read regularly that still come in printed format. And I generally read their printed version before I read the online material.

For the same reasons, I bemoan the lack of printed documentation today. Reading printed docs for tools I use rgularly is a rewarding activity. Inevitably, I find features I didn't know about. This is more difficult to do in the search-and-locate model that digital documents deliver. Seapine and Perforce are among the last of the development tools vendors to provide printed docs. And I do love them for it.

Anyway, starting on April 3, you'll have to read my InfoWorld reviews strictly on line (at www.infoworld.com). See you there!

Characterization Tests

2007-03-22T20:43:00.000-07:00

In my current column in SD Times, I discuss characterization tests, which are an as-yet little discussed form of testing. They are unit tests whose purpose is not to validate the correctness of code but to capture in tests the behavior of existing code. The idea, first popularized in Michael Feathers' excellent volume Working Effectively with Legacy Code, is that the tests can reveal the scope of any changes you make to a codebase.

You write comprehensive tests of the codebase before you touch a line. Then, you make your changes to the code, and rerun the tests. The failing tests reveal the dependencies on the code you modified. You clean up the failing tests and now they reflect the modified code base. Then, rinse, lather, repeat.

The problem historically with this approach is that writing the tests is a colossal pain, especially on large code bases. Moreover, large changes in the code break so many tests that no one wants to go back and redo 150 tests, say, just to capture the scope of changes. And so frequently, the good idea falls by the way side--with unfortunate effects when it comes time for functionally testing the revised code.To the rescue comes Agitar with a free service called JunitFactory. Get a free license and send them your Java code they will send you back the characterization tests for it. Cool idea. Change your code, run the tests, verify that nothing unexpected broke. And then have JUnitFactory re-create a complete set of characterization tests. How cool is that?

Actually, I use Agitar's service not only for characterization but for my program as I am developing it. The tests always show me unit tests I never thought of writing. Try it out. You'll like it.

MIPS per Watt: The Progression...

2007-03-18T16:29:00.000-07:00

Last week I purchased a Kill-a-Watt Electricity Usage Monitor, which measures the wattage used by any plugged-in device. It's already proving its value.

I began measuring the electrical consumption of the three workstations I use the most. The numbers strongly suggest that the power savings from multicore are real. The question that remains is whether they're substantial enough to matter to many folks, especially small sites with only a few systems. Here we go. (Notes: All workstations are Dell systems with 2GB or 3GB RAM and 2 SATA HDDs. The performance measurements are the processor arithmetic tests from the highly regarded Sandra benchmark suite. Energy consumption measured when the systems were at rest. Systems are shown in chronological order of manufacture, oldest ones first.)

The upshot is that the dual-core systems are definitely the performance equivalents of earlier dual-processor Xeon beasts (look at the performance parity between the two Intel processors), but enegy consumption of the multicore systems is almost 40% less.

However, the difference in energy consumption between the the Pentium D and the AMD system is not that great. Moreover, the difference in CPU performance, while appearing to be a lot, feels the same when I'm at the controls.

So, I think multiprocessor boxes are easy candidates for replacement by multicore systems, but upgrading multicores does not look compelling currently. (The Pentium D system is about a year older than the AMD system.)

Mylar, Tasktop, and the Intriguing Logo

2007-03-07T14:06:00.000-08:00

Yesterday, I was down at EclipseCon, a yearly gathering of the Eclipse faithful. I had lunch with Mik Kersten, the driving force behind the suddenly very popular Mylar plug-in to Eclipse that helps you organize tasks and the data that goes with them. He's working on a similar idea for desktops at his new company, Tasktop. From what I saw this will be very useful in managing the mass of data we all deal with daily. Betas are expected in late Q2.

Before you go over to the website, try to guess the meaning of Tasktop's logo: <=>

When Mik first asked me, I thought it could be dual-headed double arrow, an emoticon for a very happy, surprised person, or a vague reference to XML. But those are all wrong. Correct is: less is more. Cute!

How Many Unit Tests Are Enough?

2007-03-01T23:04:00.000-08:00

Recently, I was down visiting the folks at Agitar, who make great tools for doing unit testing. Going there always results in interesting conversations because they really live and breathe unit testing and are always finding new wrinkles in how to apply the technique. During one conversation, they casually threw out a metric for unit testing that I'd never heard before. It answers the question of: How many unit tests are enough? You'll note that pretty much all the books and essays on unit testing go to great pains to avoid answering this question, for fear, I presume, that by presenting any number they will discourage developers from writing more. Likewise, if the suggested number is high, they risk discouraging developers who will see the goal as overwhelming and unreachable.

The engineers at Agitar, however, did offer a number (as a side comment to an unrelated point). They said (I'm paraphrasing) that if the amount of test code equals the amount of program code, you're generally in pretty good shape. Their experience shows that parity of the two codebases translates into code coverage of around 70%, which means a well-tested app. Needless to say, they'd probably want to qualify this statement, but as a rule of thumb, I think it's a very practical data point--even if a bit ambitious.

I will see how it works out on my open-source projects. Currently, my ratio of test-code to app-code is (cough, cough) just over 42%. I guess I know what I'll be doing this weekend.

A Web Host Bill of Rights

2007-02-23T10:41:00.000-08:00

Like many small businesses, my company (Pacific Data Works) contracts web-hosting to a third party. Actually third parties. We have two websites, one hosted at LunarPages the other at Web.com (formerly called Interland). Our main site and mail server is at Interland.

Two days ago, Web.com suffered a massive "facilities" problem that for 10 hours shut down not only hosted accounts like ours, but Web.com itself. The company's own website was off the air.
Because of this problem, all e-mails sent to us and to other companies hosted at Web.com were bounced back as undeliverable.

Everyone understands that grave things can happen in which web-hosting services are compromised, but I don't understand what happened next: nothing. Web.com sent out no notice to customers that they might have lost e-mails or apologizing for the inconvenience. I don't care about the apology although it would be nice, but I do care about not finding out about bounced emails until I started receiving word from correspondents who were surprised their e-mails to us were rejected.

LunarPages suffered a day-long black-out last year due to a power problem in their host building. They didn't notify anyone either, but they did place a long mea-culpa on their official blog, explaining the problem. However, their website still advertises that they are down less than 9 hours a year.

I think it is about time for a Bill of Rights for customer of Web hosters. At minimum:

Web hosts should notify customers when the Web site has been unavailable for more than 4 hours.
Web hosts should notify customers when e-mail service has been down for any period in which incoming emails were bounced back.
Web hosts must post accurate information about outages on their website. An outage is defined as any period of time in which more than 20% of hosted accounts are not available.
All outages should be fully explained as to the nature of the problem and what is being done to make sure it will not recur.
Web hosts should refund the pro-rated share of hosting fees for outages automatically.

I think that's a good start.

Languages and compilation speed

2007-02-15T13:33:00.000-08:00

Yesterday, I went to visit Electric Cloud, a company that provides a suite of tools for building large applications, including a build manager and a distributed make system. While I was there I met the CEO, John Ousterhout, the designer of Tcl/Tk. He made an interesting observation about large compilations: in visits to customers with large codebases to compile, they had found that C++ code compiled the slowest, next was C, while Java was the fastest mainstream language to compile. The position of C++ does not surprise me, but Java being faster than C did--although, to be honest, I'd never given the matter much thought.

Because this discussion was a branch off the main topic, I didn't get to pursue it further, but Ousterhout did attribute it in passing to the efficiency of Java compilers. Without more info, though, I'm not sure I'm convinced that it's a question of compiler efficiency. I suspect that not needing to generate native code is a big factor, as is the simplified linking step.

Virtualization Slides

2007-02-12T20:45:00.000-08:00

I gave a pair of talks today at InfoWorld's Virtualization Executive Forum. One was pure lecture, the other a panel. The lecture (on the variety of uses of virtualization in IT) in particular was well attended. It seems managers correctly sense that there's a lot more you can do with virtualization than desktop code verification and server consolidation. Here are the slides from the lecture.

What Today's DNS Attack Looked Like

2007-02-06T23:29:00.000-08:00

As you might have read in the press today, a group of hackers tried to take down the domain name system (DNS), which is a key component of the Internet. They did this by flooding two of the top level DNS servers with requests. This picture, provided by Ripe.Net, shows the attack in full swing.

On the left are numbers 1-13, which refer to the 13 top-level DNS servers. They handle DNS requests from lower-level servers that cannot resolve a particular DNS address. As can be seen, two of the servers were targeted simultaneously in an attack that lasted several hours.

For readers who aren't familiar with DNS, it's the service that translates URLs into actual numerical addresses (which is how the Internet actually runs). So cnn.com, for example, is translated by a DNS server into 64.236.29.120. Knock out the servers that do this translation and you can only get to sites via their numerical IP addresses.

Fortunately, these 13 top-level DNS servers are redundant, so all this attack did was to slow some Internet/Web queries. However, if all 13 had been attacked, things would have become quite serious.

Learning Java via TDD: An impressive approach

2007-01-29T19:03:00.000-08:00

I have never been a fan of test-driven development. I think the concept is a curiosity that makes you write lots of throw-away code, while keeping lots of code that really doesn't test anything useful. But my most fundamental disagreement with TDD is that it is counter to the way people think. And part of the great joy of programming is building solutions, rather than building problems for the solutions to solve. For this reason, I think that many people who like TDD in the abstract eventually move back to a model of writing the code and immediately writing unit tests to exercise it. (This happens to be my model, but I didn't come to it via TDD. Rather, I adopted it because I became profoundly and deeply convinced of the value of lots of unit tests written immediately after coding.)

I've expressed this view before and would have held on to it unmodified had I not come across a terrific and unique Java tutorial: Agile Java by Jeff Langr. Forget the "agile" attribute. This is a book that teaches Java via TDD. And, it turns out, this is a very interesting way of doing things. How many times have we seen Java tomes start with "Hello World"? This one starts by teaching JUnit. And through JUnit, it teaches TDD, and then Java.

This approach, which is brilliantly original, has very distinct benefits: Firstly, readers are much more in tune with Java's verbose but unrevealing error messages. Rather than looking at a stack dump in disbelief, they're used to debugging and jumping in. Secondly, of course, they're used to testing and to writing code that is testable. The third and for me most important benefit is that this book cannot present Java in the usual way. (Such as: here's an inner class. Here's why you need it. Here's a code snippet. OK, now off to statics....) Rather, each chapter requires writing a mini-app that exercises the topics the author wants to present. So, you get to think about OO and Java, rather than merely learning language syntax.

The book also forces the reader to think about challenging testing issues. And I do mean challenging. It is the only text I've ever read anywhere that shows how to run a unit test on a log record written to the console. How would you solve that problem?

The only drawback is that the same topics are discussed in many different places throughout the book (each discussion amplifying earlier points), so you really can't use it as a reference after the fact. This is a small gripe, as there are plenty of Java references on line and in print.

I recently wrote a review of Java tutorials (before I knew about this book). Despite the many great titles I discuss in that article, if I were teaching a Java class today, this is definitely the book I would use. Bar none. And I don't even like TDD.

One activity that is inherently productive: unit testing

2007-01-27T19:39:00.000-08:00

In a post today in his blog, Larry O'Brien states: "...let's be clear that of all the things we know about software development, there are only two things that we know to be inherently highly productive: Well-treated talented programmers and iterative development incorporating client feedback"

I find it hard not to add unit testing to this list.

Of all the things that have changed in how I program during the last six or seven years, nothing comes close to unit testing in terms of making me more productive. In addition, it has made me a better programmer, because as I write code I am thinking about how to test it. And the changes that result are almost always improvements.

Multicores not as productive as you expected?

2007-01-24T19:34:00.000-08:00

For a while, I have been intrigued by how small the performance pop is from multicore processor designs. I've written about this and, finally, I think I can begin to quantify it. I'll use AMD's processors for the sole reason that the company has for years posted a performance measure for each of its processors. (Originally, this data was a move to counter Intel's now-abandoned fascination with high clock speeds.)

This chart shows the performance as published by AMD and the corresponding clock speeds for most of its recent processors. I have broken the figures out for the three branches in the Athlon processor family (which is AMD's desktop chip).

There are several interesting aspects to this chart, but the one I want to focus on is the performance of the rightmost entry. The dual-core Athon 64 X2 with a rating of 5200 has a clock speed of 2600MHz. Now, notice the Athlon XP with a rating of 2600 (10th entry from the left): it has a clock speed of 2133 MHz.

In theory, since the AMD ratings are linear, a dual-core processor should give you near but not quite the performance of two single-core chips. So two 2600-rated chips should give you roughly the performance of a 5200-rated dual core chip. Using the chart, we would expect two 2.133GHz cores to give us the 5200 performance figure. In reality, though, it takes two 2.6GHz cores to do this--far more than we would expect. It's actually an even wider gap that that, because the dual-core chips have faster buses and larger caches than the Athlon processors we're comparing it to, so it can make far better use of the processor on each clock cycle.

So, why does it take so much more than twice the clock speed to deliver dual-core performance? The orignial 2600-rated Athlon XP had a memory manager built into the chip. On the X2 chip, however, the two cores do not have dedicated memory managers--instead, they share a single on-chip memory controller. This adds overhead. The cores also share interfaces to the rest of the system and so again must work through resource contention to get the attention they need.

Don't be fooled into thinking this is an AMD-specific issue. (As I said earlier, I used AMD only because they are kind enough to publish clock speed and performance data for their chips.) This is not an AMD-only problem. Intel is in exactly the same boat--what is shared between cores is plenty expensive. Expect, as time passes, to see chip vendors trying to limit these shared resources.

Enter The Komodo Dragon

2007-01-23T16:11:00.000-08:00

Komodo 4.0 from Active State is out of beta, and was released today. Komodo has long been viewed as the premier IDE for scripting languages (Python, Perl, and Tcl especially). It continues this tradition with this release and adds Ruby and RoR support. It's the first high-end IDE with intelligent editing and debugging for Ruby and RoR that I'm aware of.

Komodo also provides numerous tools for Ajax development including Javascript debugging, specific editors for XML, HTML, and CSS. Plus an HTTP viewer and a DOM editor.

If you work in any of the languages Komodo supports, you owe it to yourself to examine it (for free). If you work in any two of them, you probably should just buy it. At $245, it's a steal.

C PDF Library

2007-01-17T19:32:00.000-08:00

In my current column in SD Times, I discuss the open-source iText library for creating PDF files. iText enables developers to create reports in PDF, HTML, and RTF from any Java application--including servlets. I mentioned in the article that I previously had looked for a PDF library in C and could not find one that was free and open source.

Eagle-eyed reader Reid Thompson kindly sent me a note pointing out the Libharu project. In my quick scan, it's not quite as feature-packed as iText, but it does cover pretty much all the functionality needed for business reports. And it has one advantage over iText that will prove compelling to some developers: Ruby bindings.

Want to be a Jolt Award Finalist? Three Steps

2007-01-16T15:25:00.000-08:00

I have been a Jolt judge for all 17 years that the award has existed. This position has been a great privilege because during that time span, the award has become the equivalent of the Academy Awards for software-development tools. One reason for this rise to prominence is that vendors sense that the judges put a lot of work and deliberation into their choices. This somewhat understates the work we do, as I'll explain shortly. This post, however, focuses on a common query from vendors whose products did not advance to the final round: what more could we have done to advance? The answer frequently is: plenty.

To give context to the pointers below, it's important to understand a few things about the Jolt judging process:

For judges, the Jolt season represents a period of intense activity. I expect that in any given Jolt season I will spend more than 100 hours on product selection. That's a lot of time, and it's all volunteer. We receive no payment for our time.
Judge deliberations are secret. We use several mechanisms for sharing our perspectives. Their contents are sealed at the end of the judging cycle. Only discussions relating to procedural matters are retained year to year. So, asking for the judges' rationale for a certain decision will not (or, at least, should not) result in useful information.
Judges vote in secret. No one but the person tabulating the results at Doctor Dobb's knows which judge voted for what. Frequently, the results are mystifying to me. I don't understand why product X was left out, while the clearly superior product Y was included. There is only one obvious answer: other people evaluate the product differently than I do.
Judges recuse themselves from any category of products in which they can derive financial benefit or where they work for one of the vendors whose product is nominated.

So, how can a vendor influence a product's fate?

Have a good product. This more than any other factor will improve your prospects. If your product is nominated year after year, make sure that you have something new to say. We frequently kick out products that are the same as last year's save for a few tweaks. Remember this is an annual award, so greatness must have occurred during the coverage year (generally November to November).
Be able to articulate why your product is better than others. Judges who have never heard of your product need a reason to vote for it. Give it to them. Many vendors set up portals specifically for Jolt judges. They include movie clips of the product (10-15 minutes), screen shots, and generated reports. This is a superb idea. Judges can go to the site and in 20 minutes figure out whether they see any magic there. If you choose to do this, emphasize how your product is different or better than others. Don't try to demo every feature. The judges just want to know why they should vote for you. If you want to create a competitive grid with feature comparisons, this helps too.
Follow up with the judges. Two categories I voted in this weekend (when we voted for finalists) had more than 30 products. After looking at a large number of websites, the products tend to blur. Even though I take notes, when I go back and re-read them, it's hard to remember my exact perspective. If I don't know a product beforehand, it's likely to fall in this blurry region unless it has some incredibly good (or bad) feature. You have a PR agency, right? Put them to work. Have them contact me. Send me a press kit in the mail. Some companies used to send 'swag'--an industry terms for those inexpensive promotional chotchkas vendors give out. Better yet, send out a boxed copy of your software. This helps. In a sea of choices, having a name to remember and with which I can associate specific features is a big plus.

Depending on how I split my votes I can vote for anywhere from 4 to 10 products per category. I almost never vote for as few as four. A problem I have is that once I've voted for the top products, I might have only a few votes left for the remaining 20-30 products. At this point, I need some reason to vote for your product. Just because it's good is not sufficient. Make its good points memorable and you're likely to get one of those last few votes.

The coming stage--choosing the winner from the finalists--is completely different. Now judges will want to download software. A common error vendors commit at this point is to make the licensing difficult. The more difficult you make the licensing, the less time I will have to look at your product. This seems obvious, but many vendors have a fear that somehow judges will do something terrible with their software--something like using it after the judging period is over. All the judges know that they can't turn over their license key to their employers. At worst, they will keep using the product for themselves. This is desirable. You want judges of the product to tell other developers, "I use the Acme debugger and it runs circles around product X." Almost invariably, the judges are influencers in major communities. Celebrate their use of your product. Make licensing easy.

I hope this post answers some preliminary questions.

Correction Re Java Tutorials

2007-01-05T17:32:00.000-08:00

Grrr. I intensely dislike being wrong in reviews and recommendations. I put lots of effort into my published writings so that readers can rely with confidence on what I recommend. So, it's quite painful--actually, it's a sense of shame and depression coupled with a lot of pain--to realize that I bollixed a review in print.

Such is the case with my quick guide to Java tutorials that appears in the current column in SD Times. In it, I write the following (after discussing the major books in this market):

Finally, for those who want something more serious but don’t require the omnibus tomes, there’s “The Java Tutorial” 4th Edition, by Zakhour et al. (Addison-Wesley Professional). In 600 pages, it presents all of the language proper, with well-chosen code examples, plus the basics of the major API sets. It’s put out by the same team that developed Sun’s outstanding online Java tutorials, which might be the best tutorials ever developed for any language. Get this book to start with, unless one of the others has a particular feature you feel is critical. Either way, you’ll be treated well.

The problem is this book is really not very good and I have to retract my recommendation. It contains some unpardonable errors (its list of Java keywords is incorrect; it refers readers to other books for topics it introduces and never brings to a close--but it only gives you the other book's title, not a chapter or page number, nor even a link to a website that covers the same topic; it makes reference to material that has not yet been presented; and, finally, it makes points that it flags as important, with no explanation as to why.). Don't consider this book, instead go to Volume 1 of Core Java to get the basics of the language in fewer than 800 pages.

My erroneous recommendation comes from an aspect that I failed to consider. The book does have good parts, especially those that refer to the latest items added to Java (enums, for example). I looked at these and was impressed. I then made the error of thinking that the rest of the book was as good as the sections I'd read. The trouble is this book is a gang project and clearly there is considerable fluctuation in the work quality of various authors. I happened to hit the few highlights in my review pass.

Curiously, a few years ago, this problem would not have occurred. Books were primarily written by one or two authors, so quality was consistent throughout--be it good or bad. Any given pair of chapters pretty much reflected the book's overall quality. With the gang books, this is much more rarely the case, as I was badly reminded.

Outsource Spam Filtering to Google

2006-12-20T12:46:00.000-08:00

A friend of mine was bending my ear the other day about the amount of spam he has to deal with. Said he, "I have three mail accounts, two of which are widely available on the Web and I am averaging more than 800 pieces of spam a day. My ISV's spam filter is OK, but I still have to check it periodically for any false positives."

He added, "I am thinking of subscribing to one of those spam services like IBM offers where I can route my mail to them. They clear out the spam, and leave the rest in my inbox."

Why pay for what you can get for free? I suggested he create an account at Google Mail, which has excellent spam detection capabilities and has never had a false negative in my experience. Then route your accounts to your Gmail account, and set up your mail agent to poll the one Gmail account. Given the 2.8GB of space Google gives you, even huge amounts of spam are not going to overflow your mailbox.

Sure enough, my friend tried it and he says in an email full of gratitude, he is free from spam.

Additional note: This solution could be done with other 'free mail' services, such as Yahoo, but I recommend doing it with Google, because:

I find Google's spam filtering is superior;

because Google resisted subpoenas to reveal user data. This last point makes me feel comfy routing my mail to them;
and because of Google's superior webmail interface, which facilitates checking mail from the road.

Give it a try.

Porting C/C++ Development from Visual Studio to Eclipse: Does that make any sense?

2006-12-14T18:34:00.000-08:00

This link is a long step-by-step tutorial on IBM Developer Works re porting C/C++ projects from Visual Studio .NET to Eclipse. You might wonder, as I did, why anyone who was using Visual Studio .NET would be tempted to port their development to a Java hosted environment to compile .NET code. I believe there is no good reason.

The article actually confirms my view , pointing out on several occasions that the Eclipse platform is indeed the wrong place for Windows development: "Neither Eclipse nor GDB understand the debugging information generated by Microsoft compilers. As a result, it is a challenge to select CDT as a full-time development environment for Windows development. However, you can use Debugging Tools for Windows for debugging side by side with Eclipse as a development environment." Ugh!

Eclipse CDT, as the article also points out, knows nothing about resources.

In other words, you'd be unlikely to ever migrate from VS.NET to Eclipse.

The article misses the opportunity to explain what Windows coding you would use the CDT for: porting code from other platforms to Windows when you don't have access to Visual Studio. That makes sense.

The new features of Java 7

2006-12-08T05:40:00.000-08:00

With Java 6 set to be released, here is a peek into Java 7. It's a PDF of a slide deck by Danny Coward, the Java SE platform lead.

Among the interesting features are: BigDecimal overloads of arithmetic operands, strings in case/switch statements, better XML and Xpath support, and the invokedynamic instruction, which facilitates the use of dynamic languages on the JVM. (Note: in my September 1 post, I mistakenly described this instruction as a feature of Java 6. It is actually in Java 7, as a commentator thoughtfully pointed out in response to that post, and revisited by this presentation.)

How much of a nerd are you, really?

2006-12-07T10:57:00.000-08:00

Some of my fans might be distressed to see that I'm a mere 59 on a scale of 100. Jeez, I barely missed a passing grade. So, what's your rating?

Convert file formats on-line

2006-12-05T18:31:00.000-08:00

This site makes it simple to convert file formats if you don't have the converter software at your easy reach--provided you're willing to send your file to a service.

Live Data Visualization at its Best

2006-11-16T23:21:00.000-08:00

This live visualization enables you to see the daily flights of FedEx planes throughout the US. Very cool and, I believe, increasingly the way data will be presented the future now that broadband is so widely available in the first world.

Mozilla Foundation Finds its Mojo

2006-11-09T21:36:00.000-08:00

Holy Toledo! The Mozilla guys are on fire! First, they released a good upgrade to the Firefox browser (despite being unable to recognize it as an upgrade from within the browser itself). Then, in the course of two days, Mozilla becomes the recipient of two major product codebases: Qualcomm's beloved Eudora mail agent and Adobe's ECMAscript execution engine. This latter project, called Tamarin, consists of the virtual machine that runs Flash Player 9.

Cool stuff! Mozilla used to be the boneyard for failed Netscape projects, but now it is suddenly at the heart of the open-source world--a pinnacle until recently occupied by the Apache Software Foundation.

As for my beloved Eudora, I need to see what happens with the project. I have been a paying customer for years. If Mozilla keeps it going, I'll stick with it. If not, I'll probably switch to Thunderbird.

Isn't Firefox 2 an Upgrade?

2006-11-03T22:50:00.001-08:00

So, isn't Firefox 2.0 an upgrade to 1.x releases? Apparently not, according to this notice when I tried to get Firefox to download the new version.

Ralph Griswold dies

2006-10-30T17:38:00.000-08:00

Ralph Griswold died earlier this month of cancer. Griswold was an important, but unsung , pioneer of programming language design. He was the inventor of SNOBOL and Icon (now Unicon), two languages distinguished by unsurpassed string processing capabilities. And I do mean unsurpassed. No language since has had such extensive native support for string operations.

The other outstanding characteristics of SNOBOL and Icon were that they were very high-level languages. This color picker was written in 31 lines of easily readable code. It might be a simple tool, but I doubt many languages could come close to that level of brevity for this same functionality (much less maintain such readability). Languages today could use a bit more of Griswold's vision, I believe.

Useful links:

Icon (which is still actively supported.)

Unicon (Icon with objects, advanced network and file functions, plus ODBC)

Griswold obituary

Solving the problem of embedded comments

2006-10-05T00:22:00.000-07:00

One problem language designers face is handling embedded comments. The basic problem is that if you comment out a large block, your intent might be thwarted by comments within the block. For example in C, the /* symbol marks the beginning of a block and */ the end. To comment out a block of code it's not sufficient to place these markers at the beginning or end of the block, because if a */ occurs inside the block, it will close the comment. So, most languages forbid the use of embedded comments. (As a result, various work-arounds are used. In C, for example, the #if 0 / #endif combo is used.)

Lua, which is a nifty dynamic language, is the first language I've seen to come up with a solution to this problem. Here goes:

In Lua, block comments start with: --[[ and end with --]] If you want to comment out a chunk that might contain block comments, you can add one or more = signs between the brackets: Open with --[=[ and close with ]=]. You can use any number of = signs between the brackets and Lua will comment out everything until it finds a pair of closing brackets with a matching number of equal signs. So, --[==[ and ]==] or --[===[ and ]===] are valid comment block markers. You can use as many equal signs as you want, as long as beginning and ending counts match. So, you can always block out your code regardless of how many embedded comments it might contain.

I think that's pretty clever. Anyone know of another language with a similar mechanism?

Interesting Uses Of Virtualization

2006-10-01T21:35:00.000-07:00

Last week, I presented two sessions at InfoWorld's Virtualization Executive Forum 2006. One was a panel on best practices for virtual test-lab automation, with representatives from VMware/Akimbi and Surgient.

The other was a solo presentation on interesting uses of virtualization--that is, compelling use cases outside of server consolidation. While hardware consolidation is still the leading driver for adoption of virtualization by IT, it will be overtaken by a slew of very interesing applications that are as yet little known. To see what I mean, have a look at the slides from my presentation. I have kept the deck short and to the point.

Amdahl's Law Revised for Hyper-Threading

2006-09-20T23:47:00.000-07:00

In the book that Rich Gerber and I wrote on Intel's Hyper-Threading Technology, we present a revised version of Amdahl's Law that reflects the constraints of Hyper-Threading Technology. There was an error in that equation that was recently reported by Gang Chen of Shanghai Jiao Tong University. The corrected formula is:

Speedup = 1 / (S + (1-S)/(0.67n) + Hn)

where S = sequential time, n = number of logical processors, H = overhead.

Clay Breshears of Intel was kind enough to double-check the revision. Ignoring overhead, if a program is 99% parallelized on a Hyper-Threading chip with two logical processors, the speedup is:

Speedup = 1 / (.01 + .99 / (.67*2)) = 1 / 0.749 = 1.34

Intel has long suggested that a 30% overall speedup was the most improvement that could realistically be achieved. The 34% improvement shown here is more theoretical than real because no overhead is included and the code is 99% parallelized. So, Intel's projections are consistent with this formula.

Thanks to Clay and to Gang Chen.

A Really Useful Ruby Book

2006-09-09T15:21:00.000-07:00

In my latest column in SD Times, I try to point out that not everything is super-cool about Ruby. One of the things I point to is the lack of good books. Sure the pick-axe book is a good place to start, but you quickly encounter the need for a practical reference that illustrates the idioms for doing standard things.

O'Reilly and Associates has just released such a book, Ruby Cookbook, (ISBN 0-596-52369-6), which contains more than 800 pages of helpful routines accompanied by thoughtful explanations. It shows how things are normally coded in Ruby. It's an impressive work especially because of its tremendous sweep: whether you need string handling, file I/O, database programming, network programming, multitasking, or accessing BitTorrent, it's all covered in this book. At a $33 street price, the book is a steal, and most Ruby developers are likely to keep a copy beside their workstation.

Other Ruby books are on the way. Most specialize on Rails, but some focus on the language as such. (Surely, there will be the bumper crop of "Ruby in 7 Days" kind of titles and other popularizing volumes, as well). So, my complaint about the lack of good books might be resolved by year end. We'll see. Meanwhile, this volume is a huge step forward.

Joel Spolsky on Writing Your Own Language

2006-09-02T17:26:00.000-07:00

Here is a neat post about why Joel's outfit wrote their own language, as well as the benefits and drawbacks of this approach.

The new bytecode in Java 6 explained

2006-09-01T14:47:00.000-07:00

For a long time, I've been looking for a cogent explanation of the new bytecode, called invokedynamic, that is being introduced in Java 6 (Mustang). This bytecode facilitates calls to scripting language methods in a context where data items are not strongly typed. Beyond that brief description, though, it's hard to find much info. Here, however, is a link that gives a good overview of the role of this bytecode and where things stand now regarding its implementation.

Review of Enerjy CQ2 (extra info)

2006-08-22T15:52:00.000-07:00

In this week's edition of InfoWorld, I review Enerjy CQ2. This is a development dashboard that integrates with existing SCM and defect tracking tools to quantify the status of a project and the productivity of developers and developer teams. I give the product a big thumbs-up, especially for Java projects, where it adds a powerful defect checker.

Unfortunately, the online version of the review does not include a screenshot, which is fairly important for seeing what the product is like. So, here is the starting page, from which you can drill down into CQ2's other dashboard windows. As you can see, it's a clean, uncluttered interface.

More on Writing Your Own Language

2006-08-21T12:03:00.000-07:00

Martin Fowler has a great series of articles on writing your own language. The advice is primarily oriented towards little languages--or what are now called domain-specific languages (DSLs). However, much of the information applies to more ambitious language projects as well.

Writing your own language -- How to choose a VM

2006-08-20T06:37:00.000-07:00

Let's say you want to write your own language, possibly even your own domain-specific language (DSL) and you want to run it on a VM. Which VM do you target?

You might instinctively think of the JVM (especially because there are so many tools to help you target the JVM) or the .NET CLR. These choices have the benefits of being highly optimized platforms on which numerous, substantial libraries already run. However in many cases, they will not be the best choice. For several reasons: they're big and difficult for users who are not technical to install (and you can't bundle them with your app); they can't be embedded into other applications; they might not support features you need.

There are many interesting alternatives that are small, reasonably fast, and have active communities supporting them.

There are, of course, the Perl and Python VMs. As you'll quickly see upon careful examination, however, both of those VMs are intimately tied to the languages they run. (No suprise there.) In addition, neither VM was really designed for targeting by other apps, so info on developing languages for them is not widely/conveniently available.

Other VMs do provide extensive support for new language developers, because they know this is the only way for them to build community. Here are three among the many you could choose from:

Lua, which consistently is the fastest performing VM outside of the JVM. It's also small, open source, widely used, and easy to embed. It's also very well documented and supported by an active community.

Neko VM, small, easy to embed, very actively supported by its developer. Particularly amenable to embedding in C applications.

Pawn, designed primarily for embedding. Its own default language is a lot like C.

These VMs are all open source. Other VM candidates are listed here, although this list is far from complete.

Most of these VMs encourage your compiler to output not bytecodes but source code using their native language. In most cases, this is the right approach because 1) the VMs can optimize your code better than you can write in their bytecode format, and 2) it saves you a whole lot of aggravation by not having to learn the bytecode and the minutia of VM internals. (Some knowldege of VM interals will, of course, be necessary.)

That being said, you do you decide which VM fits your needs best? Some principal considerations include (more in the link below):

Does the VM have native support for the data items you need?
Does the VM support the language features you need (garbage collection, multithreading, tail recursion, etc.)?
Does the VM support the performance features you need (optimization, JIT compiler, etc.)?

Once you have found a virtual machine that suits your needs, you have only to check out the community behind it, to make sure you're not nearly or completely on your own.

Note: Lambda the Ultimate is a terrific site for aficionados of programming language development. Here is a link to a post on this topic that might shed further light. If this area interest you, definitely tag/bookmark the Lambda site.

Final note: JetBrains, the makers of the popular IntelliJ IDE, have a mechanism that can be modified to provide an IDE extension for your language.

Dijkstra's Three Rules for Project Selection

2006-08-19T01:14:00.000-07:00

Edsger Dijkstra promulgated these rules in the context of scientific research, but I think they work well for selection of programming projects or, really, most intellectual endeavors that might break new ground.

The Three Golden Rules for Successful Scientific Research.

This note is devoted to three rules, the following of which is necessary if you want to be successful in scientific research. (If you manage to follow them, they will prove close to sufficient, but that is another story.) They are recorded for the benefit of those who would like to be successful in their scientific research, but fail to be so because, being unaware of these rules, they violate them. In order to avoid any misunderstanding I would like to stress, right in its first paragraph, that this note is purely pragmatic: no moral judgments are implied, and it is completely up to you to decide whether you wish to regard trying to be successful in scientific research as a noble goal in life or not. I even leave you the option of not making that decision at all.

The first rule is an "internal" one: it has nothing to do with your relation with others, it concerns you yourself in isolation. It is as follows:

"Raise your quality standards as high as you can live with, avoid wasting your time on routine problems, and always try to work as closely as possible at the boundary of your abilities. Do this, because it is the only way of discovering how that boundary should be moved forward."

This rule tells us that the obviously possible should be shunned as well as the obviously impossible: the first would not be instructive, the second would be hopeless, and both in their own way are barren.

The second rule is an "external" one: it deals with the relation between "the scientific world" and "the real world". It is as follows:

"We all like our work to be socially relevant and scientifically sound. If we can find a topic satisfying both desires, we are lucky; if the two targets are in conflict with each other, let the requirement of scientific soundness prevail."

The reason for this rule is obvious. If you do a piece of "perfect" work in which no one is interested, no harm is done, on the contrary: at least something "perfect"—be it irrelevant—has been added to our culture. If, however, you offer a shaky, would-be solution to an urgent problem, you do indeed harm to the world which, in view of the urgency of the problem, will only be too willing to apply your ineffective remedy. It is no wonder that charlatanry always flourishes in connection with incurable diseases. (Our second rule is traditionally violated by social sciences to such an extent that one can now question if they deserve the name "sciences" at all.)

The third rule is on the scale "internal/external" somewhere in between: it deals with the relation between you and your scientific colleagues. it is as follows:

"Never tackle a problem of which you can be pretty sure that (now or in the near future) it will be tackled by others who are, in relation to that problem, at least as competent and well-equipped as you."

Again the reason is obvious. If others will come up with as good a solution as you could obtain, the world doesn't loose a thing if you leave the problem alone. A corollary of the third rule is that one should never compete with one's colleagues. If you are pretty sure that in a certain area you will do a better job than anyone else, please do it in complete devotion, but when in doubt, abstain. The third rule ensures that your contributions --if any!-- will be unique.

***

I have checked the Three Golden Rules with a number of my colleagues from very different parts of the world, living and working under very different circumstances. They all agreed. And were not shocked either. The rules may strike you as a bit cruel... If so, they should, for the sooner you have discovered that the scientific world is not a soft place but--like most other worlds, for that matter--a fairly ruthless one, the better. My blessings are with you.

10 People Who Don't Matter in Tech

2006-08-18T14:04:00.000-07:00

Interesting choices (Linus Torvalds, Jonathan Schwartz, Steve Ballmer--wow) made by Business 2.0. After reading their reasons, I can't say I disagree entirely with their selections. link

VMware Workstation 5.5 Impresses

2006-01-20T10:54:00.000-08:00

For the last year or so, I have been using virtual machines (not the Java/CLR type, but the ones that enable you to run a guest operating system on a host machine) for two purposes: software reviews and long Web searches.

The use in reviews saves from me from loading and unloading software and slowly corrupting my Windows registry and dotting my win32 directory with leftover DLLs. The use in long Web surfs is to protect myself from malware. In the latter case, if your virtual machine is corrupted by malware, the entire damage is contained to the small set of files that constitute that virtual machine. You blow them away, clone an existing virtual machine, and you’re back in business. Nothing else is affected–nothing on the host system, nor any other virtual machine. Pretty cool.

Anyway, for the last year I have been using Virtual PC from Microsoft, which does a creditable job. This month, I switched to VMware Workstation 5.5. What a difference! The most conspicuous advantage of VMware is performance. When you’re running VMware at full screen, it’s hard to tell that you’re in anything but your regular, native desktop. I haven’t benchmarked it, but it’s very, very close. In contrast, with Microsoft Virtual PC 2004, you always know you’re in a virtual machine because performance is very slow. Opening dialog boxes, moving windows, starting up applications–all these actions arevisibly slow. So, not only does everything take a long time, but it is harder to assess the performance characteristics of software.

So a recommendation and a tip of the hat to VMware for a great implementation.

BEA’s AquaLogic: What is it?

2005-09-09T11:03:00.000-07:00

I met with folks from BEA earlier this week about their recently released WebLogic 9.0 application server and the AquaLogic software the company has been promoting heavily. AquaLogic, I now understand, is the company’s entry in the ESB space. It includes the ESB per se and registry functions. The ESB is based on the company’s former JMS messaging middleware, which it developed in-house. I expect I’ll be reviewing the products at some point later this year.

[Update (March 2006): I did end up writing a review of AquaLogic for InfoWorld.]

Search Engine Page Depth

2005-07-24T11:34:00.000-07:00

In my post regarding Google’s Desktop Search Engine, I mentioned several objections I had to the technology. One of them was that it did not search past 5,000 words in documents.

Subsequent research has brought to light that this partial-search is a common trait of search engines (although most of them go much farther than 5,000 words). How far down the document the indexer will go is a metric known as page depth. And, according to searchenginewatch.com, there is considerable variety in the page depth. According to the site’s 2003 figures (the latest one they have), here is how the major search engines measure up in terms of page depth:

Google: 101KB
MSN: 150 KB
Yahoo: 500 KB

So, as you see: In terms of results, a search engine only gets you partial information on what’s known to be available. Such that the absence of a result should not imply that the item does not exist. In fact, the item might even be sitting in the search engine’s document cache. However, it is simply not in the engine’s index.

Rogue Wave C++ Library now Open Source

2005-07-06T11:33:00.000-07:00

Rogue Wave (now a division of Quovadix) has open-sourced its implementation of the standard C++ library. The company gave the library to the Apache Software Foundation, which stuck it immediately in its incubator system, from which it presumably will emerge months from now. In its hey day, Rogue Wave was considered a vendor of some of the best-written libraries in C++, so I suspect quality on this will be pretty good as well. The link for more info is here.

Portable Libraries: Slim choices

2005-07-02T11:32:00.000-07:00

As I move forward with my book on Intel EM64T 64-bit extensions to x86, I want to present some complete programs. I want the code to be portable, and I don’t want to write every routine and data structure by hand. What I need is a portable library for C that can be used for free by readers. C/C++ developers with the same problem have a few choices:

For GUIs:

For everything else:

Most of what I need falls in the domain of one of the last three libraries. There are problems with each choice, however. The APR is the most actively supported of these, but it has one horrible failing: zero documentation. And because it uses a very specific model, if you don’t grok how it comes together, you will find it impossible to use.

NSPR is well documented, but is currently supported by only one person. The limitation of his time and his resources make it difficult to commit to the library with heart and soul. Although, of the three general-purpose libraries, this is probably the one that appeals to me the most.

The ACE library is another low-documentation product. In fact, other than Doxygen lists of API calls, the only documentation extant is a book. It seems to me that if you want the lib to be used a lot, the authors should make parts of the contents freely available, enough to get potential up to speed. If the library and the book are good, most every user will buy a copy. But I won’t commit to either one without some certainty going in that I will arrive satisfactorily where I want to go.

So, what to do? I will use the Qt library, which is actually much more than a GUI library. It provides support for threads, networking, sockets, database access, and XML–which are the features I need. The library is dual-licensed: free for GPL’d projects and licensed for a fee on all other projects. Since my code will be open source the free Qt solution works. Plus it’s a lovely crafted library. With great docs–not to say coverage via several books.

The oddest thing in this whole search is the APR’s lack of documentation. Despite having an active list of contributors, nobody has done docs. As a result, the only developers who use it are those who wrote or worked on the library. Seems like a stupid obstacle. I’d help out on docs if I had anyway at all to figure out how the library works, but there is literally no info.

Streaming Desktop Major League Baseball: Not Ready Yet

2005-06-29T11:21:00.000-07:00

As this year’s baseball season started up, I decided to test Major League Baseball’s new option for watching my favorite teams: streaming video of their broadcasts. A subscription costs only $14.95/month and I can watch the games of as many teams as I want. At first, this was fun, but as time passed the limitations of the subscription have led me to consider cancelling it. The problems are technological and political.

The political problem is that Major League Baseball (MLB) has exclusive national TV contracts with various networks. “Exclusive” is the operative word. For example, due to these contracts there are no streaming video of baseball games on Saturday. MLB also has regional contracts, in which a network might broadcast a game I want to watch. Even if the network doesn’t broadcast it in my home market, the mere fact that they might have precludes me from seeing it on streaming video. So, in essence, I have a 6-day/week subscription with numerous holes in the remaining schedule.

The technology problem is the video itself. It’s delivered in a 2″x3″ screen in the browser. You can make the image larger, but you lose picture quality as you expand it. The video also has a problem that it frequently stutters and then blanks out all together. In the latter case, the screen just shows the MLB logo. You have to restart the transmission to get the picture back. This happens more in the Firefox browser than IE, but it definitely happens frequently (about once per inning) in IE as well.

The bottom line is that between the technology glitches and the politics, streaming MLB video is a frustrating experience.

[Update (April 2006): I subscribed again this year and had to cancel. The service has not improved in the intervening time.]

Google Desktop Search: good, but limited

2005-06-22T11:28:00.000-07:00

Google’s enterprise desktop search (available here at no cost) is a useful tool. It does pretty much what you’d expect: it reads through your docs and e-mail and gives you an instant search capability. It has saved me several long searches for docs I knew were lying around but I couldn’t remember exactly where.

For all its benefits, however, the technology does have some distinct limitations:

it only searches the first 5,000 words of any document

you cannot remove or add items to the search database. (You can prevent it from looking in certain directories, but once its sucked up a document, you can’t remove it from its database.) The software crawls how and when it wants to. If you want it to perform a new crawl, you have only one option: uninstall it and reinstall it. No kidding!

There’s no concept of security. You cannot password-protect access to the search engine.

The engine does not search Eudora mailboxes.

The display does not highlight the searched-for keyword in the document.

For these reasons, true enterprise search tools such as IBM’s Information Integrator OmniFind do not have much to fear yet from this free offering. “Yet” is the operative word. Obviously, all these limitations are intentional, as Google has demonstrated it has the technology to get around them in its other products. So, at any time it could fold these features into its free offering.

If you can live with these limitations, you’ll definitely find the search engine useful.

Preparing for JavaOne 2005

2005-06-19T11:30:00.000-07:00

This year’s JavaOne conference is around the corner. This guide tells you how to prepare properly. Enjoy!

ISVs: Use Embedded Databases, Please

2005-06-18T11:11:00.000-07:00

This week, I completed my review of CodeAssure, a static source checker that focuses on security problems. It ships from Virginia-based Secure Software Inc. The product analyzes a C or Java codebase and generates a file of project vulnerabilities each time it’s run. This data is then shown to developers to help them locate possible security weaknesses. A separate manager’s console can compare these files, quantify project progress, and display this in a dashboard for project management purposes.

For this comparison, CodeAssure requires and enterprise DBMS (be it Oracle, DB2, or SQL Server) to store these data runs. (Up to the customer to cover the cost of obtaining the license for this.) Primarily because of this DBMS, Secure Software flies out an engineer to set up and install the package each time a customer buys a copy of CodeAssure. The question is: Why not make this database embedded? If the company used an embedded database, it could:

ship the product as a turnkey product

save the cost of sending out an installation engineer

lower their customers’ implementation costs

Those are compelling benefits.

There are some great embedded databases out there. Sleepycat software’s Berkeley DB being probably the best known and arguably the best implemented. But it’s certainly not the only choice. There are many others, some of which are completely free for commercial use, such as Apache Derby.

The reason for not using them, I believe, is that ISVs simply don’t think in terms of this kind of solution. CodeAssure is the defining example of a product where embedded databases should be used: it’s an applaction that is the sole consumer of the data it generates. I asked the Secure Software why it had not chosen the embedded model and the staff responded that they generate so much data, it could be handled only by an enterprise DBMS. This suggested to me that they had not tested embedded products sufficiently.

Embedded databases are robust and highly scalable. A case in point is Sleepycat’s product, which is thefront-end cache for Amazon.com’s product/inventory database. By size, that database is among the top 20 commercial databases in the world.

In the hybrid enterprise/embedded space, there are products like MySQL, which to my knowledge have not exhibited scalability issues.

Secure Software did allow that it was beginning to examine embedded options for future releases. I encourage them to do so, as I do all ISVs who cannot articulate a specific reason that an embedded solution is insufficient for the need. This design will make installations and deployments easier, plus save customers the cost of paying for new and expensive DBMS licenses. And so, we’ll all be a bit happier.

Choosing a programming language

2005-06-16T11:09:00.000-07:00

Programming languages rise and fall in popularity. This fact places pressure on managers to choose the right language when starting in on a big project. In my case, it places pressure on authors when choosing the language with which to exprese a concept in code. I have always chosen C, reasoning that C is readable and understandable by all C++, C#, and Java developers–which is about as broad a base of readers as I’m ever likely to reach. (Knuth’s discusses this difficult choice in the just-released Fascicle 1 of the Revision to Volume 1 of “The Art of Computer Programming” from Addison-Wesley, ISBN 0-201-85392-2)

An interesting website list the interest in specific languages based on counting Web searches for topics relating to the language. Results are normalized to avoid spurious one-off results. The methodology is explained in detail on the website. The results for the last few years are shown. I am surprised that C is still #1 (actually, it reclaimed the prize from Java a few months ago.) What is surprising is that C and Java lead C++ by such a wide margin. If I’d been asked to guess, I would have swapped the positions of C and C++ and/or possibly placed Java in first place.

There are two other interesting items. Despite rumors to the contrary, COBOL is still thriving. It’s in the most active class (those assigned a grade of ‘A’) and a fair bit ahead of Fortran and Pascal. Also, the conventional wisdom about the ‘P’ scripting languages (the three candidates for the letter P in the LAMP stack) appears to be correct. In descending order of interest, they are: Perl, PHP, and Python. Which I suspect is what most people would have guessed.

Anyway, use this info as a data point for your own decision making and in your choice of language skills.

Qt 4.0 from Trolltech

2005-06-14T11:19:00.000-07:00

I met with the folks from Trolltech last week for a presentation on Qt 4.0, which ships at quarter end. For readers unfamiliar with Qt, it is a highly portable toolkit (that is, GUI, collections, database, threading, XML functions and others) that’s used by many ISVs. It is dual-licensed, meaning that if you use it on a free, open-source project, you can get it at no cost. For anything else, you’ve got to buy a license from Trolltech. The free version is the basis for Linux’s KDE (the K Desktop Environment).

I worked with Qt 3.x for well over a year and I was very impressed. Qt is one of the cleanest libraries available. It is written C++ and is well thought out, well designed, and after a little experience it becomes intuitive to use. On Windows, I think it’s a better interface to the operating system than Win32 is. Not only because it’s better designed (You soon know the APIs without looking them up because they’re so consitently named.), but because it’s portable (to Linux, Mac, UNIX). Version 4.0 is a major upgrade. The Trolltech folks told me that they added numerous functions and also redesigned the painter so it renders far more-advanced graphics far faster. And, indeed, the demo blasted the screen.

Qt 4 also adds considerable threading support, which I think will be an important, even if underappreciated, benefit. When I was co-authoring the book on Hyper-Threading Technology for Intel Press, I had to write the same program twice. Once for Windows threads and once for Pthreads (the Linux/UNIX threading model). I really wanted a portable way to write the code. If Intel Press had allowed it, I could have used Qt 3 (I didn’t even ask because prior to Qt 4, Windows versions of the library required a paid license–there was no freebie for open-source projects.), but Qt 4 with its greater threading coverage and friendlier Windows license would be a slam-dunk today.

Qt 4 looks like an important release. (I suspect I’ll be reviewing it once it ships.) And with Macs soon migrating to Intel platforms, some code will need to be rewritten. Qt 4 probably represents a better solution than porting and tweaking old Mac code, especially for Mac developers who are entertaining any ideas of moving to Windows later.

Java IDEs Review Follow-Up

2005-05-27T11:17:00.000-07:00

My review of Java IDEs in InfoWorld generated a few corrections, which I’ll pass along here:

1) Borland JBuilder 2005: I wrote that the Ant file generation was cumbersome. This is incorrect. It can be automated with ease. (Thanks to Jeff Wilsbacher at Borland for pointing this out.)

2) IBM Rational Developer series: I stated that Rational Application Developer (aka RAD) was the RAD version and that Rational Web developer was the next level up in terms of feature completeness. But, in fact, the product named RAD (and called such inside IBM) is not the RAD tool; Rational Web Developer is. And RAD is the next tier up in terms of features. Go figure.

Those are the only corrections I’ve received.

Command line: A universal programming blindspot

2004-12-14T23:52:00.000-08:00

Every major OS today enables you to create a file whose name begins with a hyphen. Hence filenames like --help are legitimate on most systems. So is --help.class. If your class has this perfectly legal name, Java will not ever execute it. The reason is that its command-line parser (like most) is wedded to the belief that a switch character (the hyphen) will be used strictly for switches, so the parser never tests for a file name that begins with one.

I can find no parser, program or code snippet in all the code I have access to (open and closed) that checks for this possibility. Moreover, I can find no discussion of this issue. It's a true blindspot.

Of course, the problem is much greater than just Java. Most Linux and UNIX software uses the hyphen as the switch character for command-line args. Curiously, this problem does not exist in Windows, because the universal command-line character is a forward slash (/), which is not allowed in file names.

This problem points out the fact that despite the long push towards code inspection, code analysis, unit testing, etc. that has occurred during the last decade, these tools are only as good as the perceptions of the developers writing and using them. Hence, widely pervasive beliefs are still going to infect even our best efforts, until we begin to really think outside the box of our own prejudices to get at subtle aspects of software quality. I will have more on these blindspots in later posts.

SWT: Both better and lesser than Swing

2004-11-30T21:41:00.000-08:00

I've spent time these last few days coming up to speed on SWT. You know, the usual exercises of making progressively more complicated interfaces that rely on more numerous widgets with advanced features. I am struck by two things in SWT. The first is a benefit that is rarely touted: it's much simpler to code in SWT than in Swing. It takes fewer lines of code and, more importantly, you don't have to descend so deeply into the bits and bytes to be productive. Generally people point to SWT's use of native controls as its big advantage, but the easier coding is a definite benefit. For straightforward interfaces (no more complicated than Eclipse, for example), it's clearly the way to go. However, I was disappointed to find that it has no capabilities to handle rich text. It certainly handles text, but not rich text (with multiple different fonts in different sizes using different colors). JFace, the toolkit that rides above SWT, has some additional text support, but nothing close to real rich text. For this, you are obliged to use Swing. So, ultimately, I conclude that I will use SWT whenever I can, and rely on Swing for all the other applications. It's a shame really, because it would seem to me that adding the equivalent of editor toolkits to SWT would make it a feature-complete widget set.

Good book for learning JSPs and servlets

2004-11-27T21:39:00.000-08:00

As I have been studying up on servlets and JSPs, I have found one book to be particularly useful: "Head First Servlets & JSP" from O'Reilly and Associates. The whole Head First series has a sort of whimsical dimension to it. The design is slightly to entertain as well as instruct. I found this approach a little tedious in the original volume in the series "Head First Java." This book is better done and the content is provided by the two engineers who wrote Sun's certification exam on servets and JSPs. It's probably the best book around for taking you from a dead halt (knowing little or nothing about the topic) to fully proficient. Like the other Head First books, it's purely a tutorial and not meant as a reference. So, if you already know a fair bit about the topic and just need help in specific aspects of a topic, the book is probalby not for you. Otherwise, it might well be the fast, most enjoyable, and most memorable way to become great with servlets.