Wikipedia: The New Google?

The other day over lunch, x86 optimizing expert Rich Gerber articulated a change in his on-line searching that has also been showing up in my habits recently: I often search for a term in Wikipedia before I look in Google. And in many cases, my quest ends successfully in Wikipedia without recourse to Google or additional searching.

The difference is not so much in the quality of the information (although Wikipedia is demonstrably excellent), but in the quality of the links. For example, compare Wikipedia and Google results for Lua. Do Google first. At first blush, the results look pretty good. Then, search via Wikipedia. See those links, and you'll need no further convincing. (For the moment, I won't bring in the language tutorial and lots of other useful information found on the same page.)

As Gerber points out succinctly: Which would you rather have: a machine's interpolation of relevant links or links chosen by experts?

Office 2007: Getting the Hang of It

In mid-May, I decided to switch to Office 2007, fearing that I would eventually start receiving docs in the new formats and then be forced to migrate. The expected flow of docs has not materialized (except from a few contacts at Microsoft); however, I have persevered with Office 2007.

I mostly use three apps: Word (lots), Outlook (some, as I don't use it for email), and Excel (somewhat less). So, I know Word 2007 fairly well. Initially, I hated it and turned off the new silly ribbon. After heavily customizing the icon bar with the features I use most, I began finding the ribbon more useful, and now I leave it on all the time. In addition, my fingers are finding their way around the commands quickly. As time passes, my appreciation of Word 2007 deepens. There are many neat features in the user experience that are difficult to describe without doing a screencast. Suffice it to say that the new interface does eventually make you more productive and your documents more elegant. (I don't use the collaboration features very much, I should note.)

I'm going to stay with Word 2007, which I didn't expect--two weeks ago I was still cursing myself for migrating. Now that I am more productive and enjoying Word 2007, I can honestly ask myself: Would I recommend the upgrade to someone with similar needs? Probably not. The new features are certainly nice to have, but they're not compelling enough be the sole reason for migrating. Meaning that if for some reason, I were forced to switch back to Word 2003, I would manage fine-- but I would have to adjust to the absence of convenience features.

I am one of the little green men

In addition to my regular work, I have begun writing a monthly column for GreenerComputing.com, a site that specializes in discussing news and issues relating to environmental matters in computing.

The principal green issues in IT generally fall into three areas:

• power consumption
  
• regulatory compliance
  
• hardware disposal

As anyone who follows the news these days already knows, there's lots of buzz about green, so this is a fun area to working in and a good complement to my traditional areas of professional focus.

Unlike the mainstream green topics (consumer, building design, etc.), green IT makes sense only if the bottom-line aspects also make sense. So, green IT operates within narrow constraints. And because of this, it lacks the moralizing dimension and is instead purely pragmatic. I think I'll like this aspect.

Milestone 1 of Platypus shipped today

The open-source project I've been working on for months shipped its first milestone today. The project is called Platypus (for page layout and typesetting system) and it enables you to generate PDF docs (eventually HTML and RTF as well) from text files in which you embed formatting commands. It's reminiscent of TeX but updates many features from that system, adds ease of use (especially!), and eventually will add numerous report-friendly and developer-oriented features (such as language-sensitive code listings).

The current milestone is a small subset intended for early adopters who are interested enough to send feedback. It can do the following:

• basic Type 1 fonts
  
• bold, italics, underline, strikethrough
  
• foreign characters (enough for French, German, Spanish)
  
• left, center, right alignment and justified text
  
• customizable page size
  
• customizable margins
  
• indented/unindented paragraphs
  
• page-number footer
  
• debugging features
  

Milestone 2 should ship in late Q4 2007.

Documentation, examples, source code, intended schedule, and other resources are available at the project website.

Great Nerd Stim!

Writing a parser represents a kind of trophy achievement in programming, in my opinion. For years, it meant using very quirky tools such as yacc, and (later) bison. In the last few years, however, a new generation of parser generators has emerged that have eased the task considerably. Among those are CUP and JavaCC.

No tool, however, has generated more excitement than ANTLR, which is celebrated by its users for its ease of use and its ability to generate parsers in several languages. The ANTLR site has plenty of info on how to use it, but there has long been an unfilled need for a book you can bring with you to read on a long flight (and imagine the cool languages you could write, which is the real the nerd-stim).

This month, however, the pragmatic programmers filled that gap with a new book, The Definitive ANTLR Reference: Building Domain-Specific Languages . It a very readable intro to writing parsers in general, and specifically to writing ANTLR-usable language specifications. Normally, such books are long, dry, pedantic ordeals that force you to write many small tests versions before you understand enough theory to start the actual work you want to do. This book takes away much of that drudgery and makes the topic truly approachable. Actually, it sort of lulls you into the false belief that you could write a new scripting language fairly easily. Writing and implementing language specs is still not easy, but with this book and ANTLR software, it's easier than it's ever been.

Comments--How many should you have?

While there is considerable conversation about how many unit tests to write (I have two recent posts on the topic--here and here--based on conversations with vendors), few people have much to say re how many comments there should be. Once the usual themes (more comments, keep comments up-to-date, avoid pointless comments) have been stated, the conversation ends. Everyone understands what relevant and up-to-date comments mean, but few will hazard a guess as to how many of them necessary.

Interestingly, the comment ratio is a key factor in one of the most interesting metrics, the maintainability index (MI). It is also a ratio that is rewarded by Ohloh.net, the emerging site for tracking details of open-source projects. Ohloh gives projects a kudo for a high percentage of comments. The question is how high is high enough to earn the kudo. According to Ohloh, the average open source project runs around 35% comments. Projects in the top third overall get the kudo. I don't know the cut-off for this top third, but I do know Apache's FOP with a 46% ratio of comments definitely qualifies.

Comment count is a metric that is particularly easy to spoof. You could do what some OSS projects do and list all the license terms at the top of each file. I've always disliked scrolling through this pointless legalese. In my file headers, I simply point to the URL of the license, which is sufficient. But all the license boilerplate inflates comment counts amazingly. So does commenting out code and leaving it in the codebase (gag!) or writing Javadoc for simple getters and setters (also gag).

But where legitimate comments are concerned, 35% is probably a good working number to shoot for. I find many OSS projects at this ratio have quite readable code. I'll be working to bring my own codebase up to this level.

Groovy Gaining Traction

Java developers suddenly have a wealth of choices when it comes to dynamic languages that run on the JVM. There's JavaFX, which Sun announced at JavaOne this year, and JRuby, which Sun expects to complete sometime this year, and then, of course, there's my favorite: Groovy. Groovy makes writing Java programs far easier. It essentially takes Java and removes the syntactical cruft, leaving a neat language that makes you terrifically productive.

Because Groovy took a long time getting out of the gate, it's taken some licks in the press. However, it's clear that Java developers are catching on to its benefits. The JavaOne bookstore published its daily top-10 sales during the show. The picture on this post, shows the Day 2 list with two Groovy titles in the top 10 (at places 5 and 8). Overall, the Groovy bible, Groovy in Action, came in at number 5 for the show. Interest is definitely growing.

If you haven't tried Groovy yourself, it's definitely worth a look. Here are a couple of good overviews:

• The Groovy Getting Started Guide
• Andrew Glover's Overview and Tutorial on DeveloperWork

Unit Testing Private Variables and Functions

How do you write unit tests to exercise private functions and check on private variables? For my projects, I have relied on a technique of adding special testing-only methods to my classes. These methods all have names that begin with FTO_ (for testing only). My regular code may not call these functions. Eventually, I'll write a rule that code-checkers can enforce to make sure that these violations of data hiding don't accidentally appear in non-test code.

However, for a long time I've wanted to know if there is a better way to do this. So, I did what most good programmers do--I asked someone who knows testing better than I do. Which meant talking to the ever-kind Jeff Frederick, who is the main committer of the popular CI server Cruise Control (and the head of product development at Agitar).

Jeff contended that the problem is really one of code design. If all methods are short and specific, then it should be possible to test a private variable by probing the method that uses it. Or said another way: if you can't get at the variable to test it, chances are it's buried in too much code. (Extract Method, I have long believed, is the most important refactoring.)

Likewise private methods. Make 'em small, have them do only one thing, and call them from accessible methods.

I've spent a week noodling around with this sound advice. It appeals to me because almost invariably when I refactor code to make it more testable, I find that I've improved it. So far, Jeff is mostly right. I can eliminate most situations by cleaning up code. However, there are a few routines that look intractable. While I work at find a better way to refactor them (a constant quest of mine, actually), I am curious to know how you solve this problem.

Reusing IDE and SATA Drives: A Solution

Because I review lots and lots of tools, I find myself going through PCs pretty quickly. It's not fair to gauge the performance of a product on old hardware, so each year I buy new PCs. Over the course of years, I've accumulated lots of IDE drives from the PCs I've discarded. I rarely ever use them, but every once in a while I would like to know what's on them and whether I can reuse one of them. Unfortunately, this is a time consuming task, especially hooking up the drive to a PC that will access it.


I recently came across an elegant solution to this problem: the USB 2.0 Universal Drive Adapter from newertech. This device comes with a power supply for the HDD and a separate cable that plugs into an ATA-IDE drive, a notebook IDE drive, or a SATA drive. The other end of the cable is a USB plug. So, you attach the cord and the power the drive, plug the USB end of the cable into your PC--and the IDE drive magically pops up as a USB drive on your system, with full read and write capabilities.

I have cleaned up a bunch of IDE drives during the last week using this adapter. In the process, I've discovered it has some limitations. It did not work well on older drives. Some would not power up (but they did start up when I swapped them into a PC), and others did not handle read/writes well (Windows generated write errors), although it's hard to know if the errors come from the drive or the adapter. But for most drives from the last few years, the product worked without a hitch. Neat solution and, at $24.95 retail, a no-brainer purchase.

Further note: I increasingly use virtualization for my testbed infrastructure. When I'm done with a review, I archive the VMs. This keeps my disks mostly pristine and reusable, so I am not discarding disks with old PCs nearly as much as before. The fact that drives today are > 500GB also helps. ;-)

How many unit tests per method? A rule of thumb

The other day, I met with Burke Cox who heads up Stelligent, a company that specializes in helping sites set up their code-quality infrastructure (build systems, test frameworks, code coverage analysis, continuous integration--the whole works, all on a fixed-price basis). One thing Stelligent does before leaving is to impart some of the best practices they've developed over the years.

A best practice Stelligent uses for determining the number of unit tests to write for a given method struck me as completely original. The number of tests is based on the method's cyclomatic complexity (aka McCabe complexity). This complexity metric starts at 1 and adds 1 for every path the code can take. Many tools today generate cyclomatic complexity counts for methods. Stelligent's rule of thumb is:

• complexity 1-3: no test (likely the method is a getter/setter)
• complexity 3-10: 1 test
• complexity 11+: number of tests = complexity / 2

I like this guide, but would change one aspect. I think a cyclomatic complexity of 10 should have more than 1 test. I'd be more inclined to go with: complexity 3-10: # of tests = complexity / 3.

Note: you have to be careful with cyclomatic complexity. I recently wrote a switch statement that had 40 cases. Technically, that's a complexity measure of 40. Obviously, it's pointless to write lots of unit tests for 40 case statements that differ trivially. But, when the complexity number derives directly from the logic of the method, I think Stelligent's rule of thumb (with my modification) is an excellent place to start.

Effectiveness of Pair-wise Tests

The more I use unit testing, the more I wander into areas that are more typically the province of true testers (rather than of developers). One area I frequently visit is the problem of combinatorial testing, which is how to test code where there is a large number of possible values it must handle. Let's say, I have a function with four parameters that are all boolean. There are, therefore, 16 possible combos. My temptation is to write 16 unit tests. But the concept of pairwise testing argues against this test-every-permutation approach. It is based on the belief that most bugs occur between the interaction of pairs of values (rather than a specific configuration of three, or four of them). So, pair-wise experts look at the 16 possible values my switches can have and choose the minimum number in which all pairs of values have been exercised. It turns out there are 5 tests that will exercise every pair of switch combinations.

The question I've wondered about is if I write those five unit tests, rather than the more ambitious 16 tests, what have I given up? The answer is: not much. At the recent Software Test & Performance Conference, I attended a session by BJ Rollison who heads up TestingMentor.com when he's not drilling Microsoft's testers in the latest techniques. He provided some interesting figures from Microsoft's analysis of pair-wise testing.

For attrib.exe (which takes a path + 6 optional args), he did minimal testing, pairwise, and comprehensive testing with the following results:

Minimal: 9 tests, 74% code coverage, 358 code blocks covered.
Pairwise: 13 tests, 77% code coverage, 370 code blocks covered
Maximal: 972 tests, 77% code coverage, 370 code blocks covered

A similar test exploration with findstr.exe (which takes a string + 19 optional args) found that pairwise testing via 136 tests covered 74% of the app, while maximal coverage consisting of 3,533 tests covered 76% of the app.

These numbers make sense. Surely, if you test a key subset of pairs possibilities, testing additional combinations is not likely to exercise completely different routines, so code coverage should not increase much for the tests that exceed pair-wise recommendations. What surprised me was that pairwise got such high-numbers to begin with. 70+ % is pretty decent coverage.

From now on, pair-wise testing will be part of my unit-testing design toolbox. For a list of tools that can find the pairs to test, see here. Rollison highly recommended Microsoft's free PICT tool (see previous link), which also provides a means to specify special relationships between the various factors in the combinations.

Update to my review of Java IDEs

My review of three leading enterprise Java IDEs appeared in later March in InfoWorld. I've received some nice comments on the piece and a few corrections. Most of the corrections come from Sun in my coverage of NetBeans 5.5. Here are the principal items:

• I complained that NetBeans does not use anti-aliased fonts. I overlooked a switch that can turn on these fonts in Java 5. On Java 6, they're on by default, if your system is set with font smoothing on. (It's hard to figure why NetBeans does not default to these fonts on Java 5, as do Eclipse, IntelliJ, and most of the other IDEs.)
  
• I recommended that potential users look at NetBeans 6.0 beta, because it adds many of the features that I complain are missing. Sun gently points out that the current version of 6.0 is not quite in beta yet, but should be in beta within the next few months. For the latest version and roadmap, go to netbeans.org
• After extensive discussions, Sun convinced me that they have greater support for Java 6 than I originally gave them credit for. I originally wrote they provided 'minimal' coverage. In retrospect, I would say 'good' support for Java 6.
  
• Finally, Sun was kind enough to point out that I give them too much credit in saying that they have deployment support for JavaDB. In fact, they provide only start/stop control from within NetBeans.
  

If there are further corrections, I'll update this post.

InfoWorld Moves to Online-Only

InfoWorld magazine announced today that it would be abandoning the print publication and going to an entirely online format. This move continues a trend that has been in place for many years in technical publications. I understand the economics of the move and think that on that basis, it's the right move. However, I confess some sadness at this transition. I like print media. I am currently overseas. And it was a great pleasure to pack magazines with my carry-on luggage and read through pages of material on the plane. SD Times and Doctor Dobb's are the only developer mags I read regularly that still come in printed format. And I generally read their printed version before I read the online material.

For the same reasons, I bemoan the lack of printed documentation today. Reading printed docs for tools I use rgularly is a rewarding activity. Inevitably, I find features I didn't know about. This is more difficult to do in the search-and-locate model that digital documents deliver. Seapine and Perforce are among the last of the development tools vendors to provide printed docs. And I do love them for it.

Anyway, starting on April 3, you'll have to read my InfoWorld reviews strictly on line (at www.infoworld.com). See you there!

Characterization Tests

In my current column in SD Times, I discuss characterization tests, which are an as-yet little discussed form of testing. They are unit tests whose purpose is not to validate the correctness of code but to capture in tests the behavior of existing code. The idea, first popularized in Michael Feathers' excellent volume Working Effectively with Legacy Code, is that the tests can reveal the scope of any changes you make to a codebase.

You write comprehensive tests of the codebase before you touch a line. Then, you make your changes to the code, and rerun the tests. The failing tests reveal the dependencies on the code you modified. You clean up the failing tests and now they reflect the modified code base. Then, rinse, lather, repeat.

The problem historically with this approach is that writing the tests is a colossal pain, especially on large code bases. Moreover, large changes in the code break so many tests that no one wants to go back and redo 150 tests, say, just to capture the scope of changes. And so frequently, the good idea falls by the way side--with unfortunate effects when it comes time for functionally testing the revised code.To the rescue comes Agitar with a free service called JunitFactory. Get a free license and send them your Java code they will send you back the characterization tests for it. Cool idea. Change your code, run the tests, verify that nothing unexpected broke. And then have JUnitFactory re-create a complete set of characterization tests. How cool is that?

Actually, I use Agitar's service not only for characterization but for my program as I am developing it. The tests always show me unit tests I never thought of writing. Try it out. You'll like it.

MIPS per Watt: The Progression...

Last week I purchased a Kill-a-Watt Electricity Usage Monitor, which measures the wattage used by any plugged-in device. It's already proving its value.

I began measuring the electrical consumption of the three workstations I use the most. The numbers strongly suggest that the power savings from multicore are real. The question that remains is whether they're substantial enough to matter to many folks, especially small sites with only a few systems. Here we go. (Notes: All workstations are Dell systems with 2GB or 3GB RAM and 2 SATA HDDs. The performance measurements are the processor arithmetic tests from the highly regarded Sandra benchmark suite. Energy consumption measured when the systems were at rest. Systems are shown in chronological order of manufacture, oldest ones first.)


The upshot is that the dual-core systems are definitely the performance equivalents of earlier dual-processor Xeon beasts (look at the performance parity between the two Intel processors), but enegy consumption of the multicore systems is almost 40% less.

However, the difference in energy consumption between the the Pentium D and the AMD system is not that great. Moreover, the difference in CPU performance, while appearing to be a lot, feels the same when I'm at the controls.

So, I think multiprocessor boxes are easy candidates for replacement by multicore systems, but upgrading multicores does not look compelling currently. (The Pentium D system is about a year older than the AMD system.)

Mylar, Tasktop, and the Intriguing Logo

Yesterday, I was down at EclipseCon, a yearly gathering of the Eclipse faithful. I had lunch with Mik Kersten, the driving force behind the suddenly very popular Mylar plug-in to Eclipse that helps you organize tasks and the data that goes with them. He's working on a similar idea for desktops at his new company, Tasktop. From what I saw this will be very useful in managing the mass of data we all deal with daily. Betas are expected in late Q2.

Before you go over to the website, try to guess the meaning of Tasktop's logo: <=>

When Mik first asked me, I thought it could be dual-headed double arrow, an emoticon for a very happy, surprised person, or a vague reference to XML. But those are all wrong. Correct is: less is more. Cute!

How Many Unit Tests Are Enough?

Recently, I was down visiting the folks at Agitar, who make great tools for doing unit testing. Going there always results in interesting conversations because they really live and breathe unit testing and are always finding new wrinkles in how to apply the technique. During one conversation, they casually threw out a metric for unit testing that I'd never heard before. It answers the question of: How many unit tests are enough? You'll note that pretty much all the books and essays on unit testing go to great pains to avoid answering this question, for fear, I presume, that by presenting any number they will discourage developers from writing more. Likewise, if the suggested number is high, they risk discouraging developers who will see the goal as overwhelming and unreachable.

The engineers at Agitar, however, did offer a number (as a side comment to an unrelated point). They said (I'm paraphrasing) that if the amount of test code equals the amount of program code, you're generally in pretty good shape. Their experience shows that parity of the two codebases translates into code coverage of around 70%, which means a well-tested app. Needless to say, they'd probably want to qualify this statement, but as a rule of thumb, I think it's a very practical data point--even if a bit ambitious.

I will see how it works out on my open-source projects. Currently, my ratio of test-code to app-code is (cough, cough) just over 42%. I guess I know what I'll be doing this weekend.

A Web Host Bill of Rights

Like many small businesses, my company (Pacific Data Works) contracts web-hosting to a third party. Actually third parties. We have two websites, one hosted at LunarPages the other at Web.com (formerly called Interland). Our main site and mail server is at Interland.

Two days ago, Web.com suffered a massive "facilities" problem that for 10 hours shut down not only hosted accounts like ours, but Web.com itself. The company's own website was off the air.
Because of this problem, all e-mails sent to us and to other companies hosted at Web.com were bounced back as undeliverable.

Everyone understands that grave things can happen in which web-hosting services are compromised, but I don't understand what happened next: nothing. Web.com sent out no notice to customers that they might have lost e-mails or apologizing for the inconvenience. I don't care about the apology although it would be nice, but I do care about not finding out about bounced emails until I started receiving word from correspondents who were surprised their e-mails to us were rejected.

LunarPages suffered a day-long black-out last year due to a power problem in their host building. They didn't notify anyone either, but they did place a long mea-culpa on their official blog, explaining the problem. However, their website still advertises that they are down less than 9 hours a year.

I think it is about time for a Bill of Rights for customer of Web hosters. At minimum:

• Web hosts should notify customers when the Web site has been unavailable for more than 4 hours.
• Web hosts should notify customers when e-mail service has been down for any period in which incoming emails were bounced back.
• Web hosts must post accurate information about outages on their website. An outage is defined as any period of time in which more than 20% of hosted accounts are not available.
• All outages should be fully explained as to the nature of the problem and what is being done to make sure it will not recur.
• Web hosts should refund the pro-rated share of hosting fees for outages automatically.

I think that's a good start.

Languages and compilation speed

Yesterday, I went to visit Electric Cloud, a company that provides a suite of tools for building large applications, including a build manager and a distributed make system. While I was there I met the CEO, John Ousterhout, the designer of Tcl/Tk. He made an interesting observation about large compilations: in visits to customers with large codebases to compile, they had found that C++ code compiled the slowest, next was C, while Java was the fastest mainstream language to compile. The position of C++ does not surprise me, but Java being faster than C did--although, to be honest, I'd never given the matter much thought.

Because this discussion was a branch off the main topic, I didn't get to pursue it further, but Ousterhout did attribute it in passing to the efficiency of Java compilers. Without more info, though, I'm not sure I'm convinced that it's a question of compiler efficiency. I suspect that not needing to generate native code is a big factor, as is the simplified linking step.

Virtualization Slides

I gave a pair of talks today at InfoWorld's Virtualization Executive Forum. One was pure lecture, the other a panel. The lecture (on the variety of uses of virtualization in IT) in particular was well attended. It seems managers correctly sense that there's a lot more you can do with virtualization than desktop code verification and server consolidation. Here are the slides from the lecture.

What Today's DNS Attack Looked Like

As you might have read in the press today, a group of hackers tried to take down the domain name system (DNS), which is a key component of the Internet. They did this by flooding two of the top level DNS servers with requests. This picture, provided by Ripe.Net, shows the attack in full swing.





On the left are numbers 1-13, which refer to the 13 top-level DNS servers. They handle DNS requests from lower-level servers that cannot resolve a particular DNS address. As can be seen, two of the servers were targeted simultaneously in an attack that lasted several hours.

For readers who aren't familiar with DNS, it's the service that translates URLs into actual numerical addresses (which is how the Internet actually runs). So cnn.com, for example, is translated by a DNS server into 64.236.29.120. Knock out the servers that do this translation and you can only get to sites via their numerical IP addresses.

Fortunately, these 13 top-level DNS servers are redundant, so all this attack did was to slow some Internet/Web queries. However, if all 13 had been attacked, things would have become quite serious.

Learning Java via TDD: An impressive approach

I have never been a fan of test-driven development. I think the concept is a curiosity that makes you write lots of throw-away code, while keeping lots of code that really doesn't test anything useful. But my most fundamental disagreement with TDD is that it is counter to the way people think. And part of the great joy of programming is building solutions, rather than building problems for the solutions to solve. For this reason, I think that many people who like TDD in the abstract eventually move back to a model of writing the code and immediately writing unit tests to exercise it. (This happens to be my model, but I didn't come to it via TDD. Rather, I adopted it because I became profoundly and deeply convinced of the value of lots of unit tests written immediately after coding.)

I've expressed this view before and would have held on to it unmodified had I not come across a terrific and unique Java tutorial: Agile Java by Jeff Langr. Forget the "agile" attribute. This is a book that teaches Java via TDD. And, it turns out, this is a very interesting way of doing things. How many times have we seen Java tomes start with "Hello World"? This one starts by teaching JUnit. And through JUnit, it teaches TDD, and then Java.

This approach, which is brilliantly original, has very distinct benefits: Firstly, readers are much more in tune with Java's verbose but unrevealing error messages. Rather than looking at a stack dump in disbelief, they're used to debugging and jumping in. Secondly, of course, they're used to testing and to writing code that is testable. The third and for me most important benefit is that this book cannot present Java in the usual way. (Such as: here's an inner class. Here's why you need it. Here's a code snippet. OK, now off to statics....) Rather, each chapter requires writing a mini-app that exercises the topics the author wants to present. So, you get to think about OO and Java, rather than merely learning language syntax.

The book also forces the reader to think about challenging testing issues. And I do mean challenging. It is the only text I've ever read anywhere that shows how to run a unit test on a log record written to the console. How would you solve that problem?

The only drawback is that the same topics are discussed in many different places throughout the book (each discussion amplifying earlier points), so you really can't use it as a reference after the fact. This is a small gripe, as there are plenty of Java references on line and in print.

I recently wrote a review of Java tutorials (before I knew about this book). Despite the many great titles I discuss in that article, if I were teaching a Java class today, this is definitely the book I would use. Bar none. And I don't even like TDD.

One activity that is inherently productive: unit testing

In a post today in his blog, Larry O'Brien states: "...let's be clear that of all the things we know about software development, there are only two things that we know to be inherently highly productive: Well-treated talented programmers and iterative development incorporating client feedback"

I find it hard not to add unit testing to this list.

Of all the things that have changed in how I program during the last six or seven years, nothing comes close to unit testing in terms of making me more productive. In addition, it has made me a better programmer, because as I write code I am thinking about how to test it. And the changes that result are almost always improvements.

Multicores not as productive as you expected?

For a while, I have been intrigued by how small the performance pop is from multicore processor designs. I've written about this and, finally, I think I can begin to quantify it. I'll use AMD's processors for the sole reason that the company has for years posted a performance measure for each of its processors. (Originally, this data was a move to counter Intel's now-abandoned fascination with high clock speeds.)

This chart shows the performance as published by AMD and the corresponding clock speeds for most of its recent processors. I have broken the figures out for the three branches in the Athlon processor family (which is AMD's desktop chip).

There are several interesting aspects to this chart, but the one I want to focus on is the performance of the rightmost entry. The dual-core Athon 64 X2 with a rating of 5200 has a clock speed of 2600MHz. Now, notice the Athlon XP with a rating of 2600 (10th entry from the left): it has a clock speed of 2133 MHz.

In theory, since the AMD ratings are linear, a dual-core processor should give you near but not quite the performance of two single-core chips. So two 2600-rated chips should give you roughly the performance of a 5200-rated dual core chip. Using the chart, we would expect two 2.133GHz cores to give us the 5200 performance figure. In reality, though, it takes two 2.6GHz cores to do this--far more than we would expect. It's actually an even wider gap that that, because the dual-core chips have faster buses and larger caches than the Athlon processors we're comparing it to, so it can make far better use of the processor on each clock cycle.

So, why does it take so much more than twice the clock speed to deliver dual-core performance? The orignial 2600-rated Athlon XP had a memory manager built into the chip. On the X2 chip, however, the two cores do not have dedicated memory managers--instead, they share a single on-chip memory controller. This adds overhead. The cores also share interfaces to the rest of the system and so again must work through resource contention to get the attention they need.

Don't be fooled into thinking this is an AMD-specific issue. (As I said earlier, I used AMD only because they are kind enough to publish clock speed and performance data for their chips.) This is not an AMD-only problem. Intel is in exactly the same boat--what is shared between cores is plenty expensive. Expect, as time passes, to see chip vendors trying to limit these shared resources.

Enter The Komodo Dragon

Komodo 4.0 from Active State is out of beta, and was released today. Komodo has long been viewed as the premier IDE for scripting languages (Python, Perl, and Tcl especially). It continues this tradition with this release and adds Ruby and RoR support. It's the first high-end IDE with intelligent editing and debugging for Ruby and RoR that I'm aware of.

Komodo also provides numerous tools for Ajax development including Javascript debugging, specific editors for XML, HTML, and CSS. Plus an HTTP viewer and a DOM editor.

If you work in any of the languages Komodo supports, you owe it to yourself to examine it (for free). If you work in any two of them, you probably should just buy it. At $245, it's a steal.

C PDF Library

In my current column in SD Times, I discuss the open-source iText library for creating PDF files. iText enables developers to create reports in PDF, HTML, and RTF from any Java application--including servlets. I mentioned in the article that I previously had looked for a PDF library in C and could not find one that was free and open source.

Eagle-eyed reader Reid Thompson kindly sent me a note pointing out the Libharu project. In my quick scan, it's not quite as feature-packed as iText, but it does cover pretty much all the functionality needed for business reports. And it has one advantage over iText that will prove compelling to some developers: Ruby bindings.

Want to be a Jolt Award Finalist? Three Steps

I have been a Jolt judge for all 17 years that the award has existed. This position has been a great privilege because during that time span, the award has become the equivalent of the Academy Awards for software-development tools. One reason for this rise to prominence is that vendors sense that the judges put a lot of work and deliberation into their choices. This somewhat understates the work we do, as I'll explain shortly. This post, however, focuses on a common query from vendors whose products did not advance to the final round: what more could we have done to advance? The answer frequently is: plenty.

To give context to the pointers below, it's important to understand a few things about the Jolt judging process:

• For judges, the Jolt season represents a period of intense activity. I expect that in any given Jolt season I will spend more than 100 hours on product selection. That's a lot of time, and it's all volunteer. We receive no payment for our time.
  
  
• Judge deliberations are secret. We use several mechanisms for sharing our perspectives. Their contents are sealed at the end of the judging cycle. Only discussions relating to procedural matters are retained year to year. So, asking for the judges' rationale for a certain decision will not (or, at least, should not) result in useful information.
  
  
• Judges vote in secret. No one but the person tabulating the results at Doctor Dobb's knows which judge voted for what. Frequently, the results are mystifying to me. I don't understand why product X was left out, while the clearly superior product Y was included. There is only one obvious answer: other people evaluate the product differently than I do.
  
  
• Judges recuse themselves from any category of products in which they can derive financial benefit or where they work for one of the vendors whose product is nominated.

So, how can a vendor influence a product's fate?

1. Have a good product. This more than any other factor will improve your prospects. If your product is nominated year after year, make sure that you have something new to say. We frequently kick out products that are the same as last year's save for a few tweaks. Remember this is an annual award, so greatness must have occurred during the coverage year (generally November to November).
   
   
2. Be able to articulate why your product is better than others. Judges who have never heard of your product need a reason to vote for it. Give it to them. Many vendors set up portals specifically for Jolt judges. They include movie clips of the product (10-15 minutes), screen shots, and generated reports. This is a superb idea. Judges can go to the site and in 20 minutes figure out whether they see any magic there. If you choose to do this, emphasize how your product is different or better than others. Don't try to demo every feature. The judges just want to know why they should vote for you. If you want to create a competitive grid with feature comparisons, this helps too.
   
   
3. Follow up with the judges. Two categories I voted in this weekend (when we voted for finalists) had more than 30 products. After looking at a large number of websites, the products tend to blur. Even though I take notes, when I go back and re-read them, it's hard to remember my exact perspective. If I don't know a product beforehand, it's likely to fall in this blurry region unless it has some incredibly good (or bad) feature. You have a PR agency, right? Put them to work. Have them contact me. Send me a press kit in the mail. Some companies used to send 'swag'--an industry terms for those inexpensive promotional chotchkas vendors give out. Better yet, send out a boxed copy of your software. This helps. In a sea of choices, having a name to remember and with which I can associate specific features is a big plus.
   
   Depending on how I split my votes I can vote for anywhere from 4 to 10 products per category. I almost never vote for as few as four. A problem I have is that once I've voted for the top products, I might have only a few votes left for the remaining 20-30 products. At this point, I need some reason to vote for your product. Just because it's good is not sufficient. Make its good points memorable and you're likely to get one of those last few votes.
   

The coming stage--choosing the winner from the finalists--is completely different. Now judges will want to download software. A common error vendors commit at this point is to make the licensing difficult. The more difficult you make the licensing, the less time I will have to look at your product. This seems obvious, but many vendors have a fear that somehow judges will do something terrible with their software--something like using it after the judging period is over. All the judges know that they can't turn over their license key to their employers. At worst, they will keep using the product for themselves. This is desirable. You want judges of the product to tell other developers, "I use the Acme debugger and it runs circles around product X." Almost invariably, the judges are influencers in major communities. Celebrate their use of your product. Make licensing easy.

I hope this post answers some preliminary questions.

Correction Re Java Tutorials

Grrr. I intensely dislike being wrong in reviews and recommendations. I put lots of effort into my published writings so that readers can rely with confidence on what I recommend. So, it's quite painful--actually, it's a sense of shame and depression coupled with a lot of pain--to realize that I bollixed a review in print.

Such is the case with my quick guide to Java tutorials that appears in the current column in SD Times. In it, I write the following (after discussing the major books in this market):

Finally, for those who want something more serious but don’t require the omnibus tomes, there’s “The Java Tutorial” 4th Edition, by Zakhour et al. (Addison-Wesley Professional). In 600 pages, it presents all of the language proper, with well-chosen code examples, plus the basics of the major API sets. It’s put out by the same team that developed Sun’s outstanding online Java tutorials, which might be the best tutorials ever developed for any language. Get this book to start with, unless one of the others has a particular feature you feel is critical. Either way, you’ll be treated well.

The problem is this book is really not very good and I have to retract my recommendation. It contains some unpardonable errors (its list of Java keywords is incorrect; it refers readers to other books for topics it introduces and never brings to a close--but it only gives you the other book's title, not a chapter or page number, nor even a link to a website that covers the same topic; it makes reference to material that has not yet been presented; and, finally, it makes points that it flags as important, with no explanation as to why.). Don't consider this book, instead go to Volume 1 of Core Java to get the basics of the language in fewer than 800 pages.

My erroneous recommendation comes from an aspect that I failed to consider. The book does have good parts, especially those that refer to the latest items added to Java (enums, for example). I looked at these and was impressed. I then made the error of thinking that the rest of the book was as good as the sections I'd read. The trouble is this book is a gang project and clearly there is considerable fluctuation in the work quality of various authors. I happened to hit the few highlights in my review pass.

Curiously, a few years ago, this problem would not have occurred. Books were primarily written by one or two authors, so quality was consistent throughout--be it good or bad. Any given pair of chapters pretty much reflected the book's overall quality. With the gang books, this is much more rarely the case, as I was badly reminded.

Outsource Spam Filtering to Google

A friend of mine was bending my ear the other day about the amount of spam he has to deal with. Said he, "I have three mail accounts, two of which are widely available on the Web and I am averaging more than 800 pieces of spam a day. My ISV's spam filter is OK, but I still have to check it periodically for any false positives."

He added, "I am thinking of subscribing to one of those spam services like IBM offers where I can route my mail to them. They clear out the spam, and leave the rest in my inbox."

Why pay for what you can get for free? I suggested he create an account at Google Mail, which has excellent spam detection capabilities and has never had a false negative in my experience. Then route your accounts to your Gmail account, and set up your mail agent to poll the one Gmail account. Given the 2.8GB of space Google gives you, even huge amounts of spam are not going to overflow your mailbox.

Sure enough, my friend tried it and he says in an email full of gratitude, he is free from spam.

Additional note: This solution could be done with other 'free mail' services, such as Yahoo, but I recommend doing it with Google, because:

• I find Google's spam filtering is superior;

• because Google resisted subpoenas to reveal user data. This last point makes me feel comfy routing my mail to them;
• and because of Google's superior webmail interface, which facilitates checking mail from the road.

Give it a try.

Porting C/C++ Development from Visual Studio to Eclipse: Does that make any sense?

This link is a long step-by-step tutorial on IBM Developer Works re porting C/C++ projects from Visual Studio .NET to Eclipse. You might wonder, as I did, why anyone who was using Visual Studio .NET would be tempted to port their development to a Java hosted environment to compile .NET code. I believe there is no good reason.

The article actually confirms my view , pointing out on several occasions that the Eclipse platform is indeed the wrong place for Windows development: "Neither Eclipse nor GDB understand the debugging information generated by Microsoft compilers. As a result, it is a challenge to select CDT as a full-time development environment for Windows development. However, you can use Debugging Tools for Windows for debugging side by side with Eclipse as a development environment." Ugh!

Eclipse CDT, as the article also points out, knows nothing about resources.

In other words, you'd be unlikely to ever migrate from VS.NET to Eclipse.

The article misses the opportunity to explain what Windows coding you would use the CDT for: porting code from other platforms to Windows when you don't have access to Visual Studio. That makes sense.

The new features of Java 7

With Java 6 set to be released, here is a peek into Java 7. It's a PDF of a slide deck by Danny Coward, the Java SE platform lead.

Among the interesting features are: BigDecimal overloads of arithmetic operands, strings in case/switch statements, better XML and Xpath support, and the invokedynamic instruction, which facilitates the use of dynamic languages on the JVM. (Note: in my September 1 post, I mistakenly described this instruction as a feature of Java 6. It is actually in Java 7, as a commentator thoughtfully pointed out in response to that post, and revisited by this presentation.)

How much of a nerd are you, really?

Some of my fans might be distressed to see that I'm a mere 59 on a scale of 100. Jeez, I barely missed a passing grade. So, what's your rating?

Convert file formats on-line

This site makes it simple to convert file formats if you don't have the converter software at your easy reach--provided you're willing to send your file to a service.

Live Data Visualization at its Best

This live visualization enables you to see the daily flights of FedEx planes throughout the US. Very cool and, I believe, increasingly the way data will be presented the future now that broadband is so widely available in the first world.

Mozilla Foundation Finds its Mojo

Holy Toledo! The Mozilla guys are on fire! First, they released a good upgrade to the Firefox browser (despite being unable to recognize it as an upgrade from within the browser itself). Then, in the course of two days, Mozilla becomes the recipient of two major product codebases: Qualcomm's beloved Eudora mail agent and Adobe's ECMAscript execution engine. This latter project, called Tamarin, consists of the virtual machine that runs Flash Player 9.

Cool stuff! Mozilla used to be the boneyard for failed Netscape projects, but now it is suddenly at the heart of the open-source world--a pinnacle until recently occupied by the Apache Software Foundation.

As for my beloved Eudora, I need to see what happens with the project. I have been a paying customer for years. If Mozilla keeps it going, I'll stick with it. If not, I'll probably switch to Thunderbird.

Isn't Firefox 2 an Upgrade?

So, isn't Firefox 2.0 an upgrade to 1.x releases? Apparently not, according to this notice when I tried to get Firefox to download the new version.

Ralph Griswold dies

Ralph Griswold died earlier this month of cancer. Griswold was an important, but unsung , pioneer of programming language design. He was the inventor of SNOBOL and Icon (now Unicon), two languages distinguished by unsurpassed string processing capabilities. And I do mean unsurpassed. No language since has had such extensive native support for string operations.

The other outstanding characteristics of SNOBOL and Icon were that they were very high-level languages. This color picker was written in 31 lines of easily readable code. It might be a simple tool, but I doubt many languages could come close to that level of brevity for this same functionality (much less maintain such readability). Languages today could use a bit more of Griswold's vision, I believe.

Useful links:

Icon (which is still actively supported.)

Unicon (Icon with objects, advanced network and file functions, plus ODBC)

Griswold obituary

Solving the problem of embedded comments

One problem language designers face is handling embedded comments. The basic problem is that if you comment out a large block, your intent might be thwarted by comments within the block. For example in C, the /* symbol marks the beginning of a block and */ the end. To comment out a block of code it's not sufficient to place these markers at the beginning or end of the block, because if a */ occurs inside the block, it will close the comment. So, most languages forbid the use of embedded comments. (As a result, various work-arounds are used. In C, for example, the #if 0 / #endif combo is used.)

Lua, which is a nifty dynamic language, is the first language I've seen to come up with a solution to this problem. Here goes:

In Lua, block comments start with: --[[ and end with --]] If you want to comment out a chunk that might contain block comments, you can add one or more = signs between the brackets: Open with --[=[ and close with ]=]. You can use any number of = signs between the brackets and Lua will comment out everything until it finds a pair of closing brackets with a matching number of equal signs. So, --[==[ and ]==] or --[===[ and ]===] are valid comment block markers. You can use as many equal signs as you want, as long as beginning and ending counts match. So, you can always block out your code regardless of how many embedded comments it might contain.

I think that's pretty clever. Anyone know of another language with a similar mechanism?

Interesting Uses Of Virtualization

Last week, I presented two sessions at InfoWorld's Virtualization Executive Forum 2006. One was a panel on best practices for virtual test-lab automation, with representatives from VMware/Akimbi and Surgient.

The other was a solo presentation on interesting uses of virtualization--that is, compelling use cases outside of server consolidation. While hardware consolidation is still the leading driver for adoption of virtualization by IT, it will be overtaken by a slew of very interesing applications that are as yet little known. To see what I mean, have a look at the slides from my presentation. I have kept the deck short and to the point.

Amdahl's Law Revised for Hyper-Threading

In the book that Rich Gerber and I wrote on Intel's Hyper-Threading Technology, we present a revised version of Amdahl's Law that reflects the constraints of Hyper-Threading Technology. There was an error in that equation that was recently reported by Gang Chen of Shanghai Jiao Tong University. The corrected formula is:

Speedup = 1 / (S + (1-S)/(0.67n) + Hn)

where S = sequential time, n = number of logical processors, H = overhead.

Clay Breshears of Intel was kind enough to double-check the revision. Ignoring overhead, if a program is 99% parallelized on a Hyper-Threading chip with two logical processors, the speedup is:

Speedup = 1 / (.01 + .99 / (.67*2)) = 1 / 0.749 = 1.34

Intel has long suggested that a 30% overall speedup was the most improvement that could realistically be achieved. The 34% improvement shown here is more theoretical than real because no overhead is included and the code is 99% parallelized. So, Intel's projections are consistent with this formula.

Thanks to Clay and to Gang Chen.

A Really Useful Ruby Book

In my latest column in SD Times, I try to point out that not everything is super-cool about Ruby. One of the things I point to is the lack of good books. Sure the pick-axe book is a good place to start, but you quickly encounter the need for a practical reference that illustrates the idioms for doing standard things.

O'Reilly and Associates has just released such a book, Ruby Cookbook, (ISBN 0-596-52369-6), which contains more than 800 pages of helpful routines accompanied by thoughtful explanations. It shows how things are normally coded in Ruby. It's an impressive work especially because of its tremendous sweep: whether you need string handling, file I/O, database programming, network programming, multitasking, or accessing BitTorrent, it's all covered in this book. At a $33 street price, the book is a steal, and most Ruby developers are likely to keep a copy beside their workstation.

Other Ruby books are on the way. Most specialize on Rails, but some focus on the language as such. (Surely, there will be the bumper crop of "Ruby in 7 Days" kind of titles and other popularizing volumes, as well). So, my complaint about the lack of good books might be resolved by year end. We'll see. Meanwhile, this volume is a huge step forward.

Joel Spolsky on Writing Your Own Language

Here is a neat post about why Joel's outfit wrote their own language, as well as the benefits and drawbacks of this approach.

The new bytecode in Java 6 explained

For a long time, I've been looking for a cogent explanation of the new bytecode, called invokedynamic, that is being introduced in Java 6 (Mustang). This bytecode facilitates calls to scripting language methods in a context where data items are not strongly typed. Beyond that brief description, though, it's hard to find much info. Here, however, is a link that gives a good overview of the role of this bytecode and where things stand now regarding its implementation.

Review of Enerjy CQ2 (extra info)

In this week's edition of InfoWorld, I review Enerjy CQ2. This is a development dashboard that integrates with existing SCM and defect tracking tools to quantify the status of a project and the productivity of developers and developer teams. I give the product a big thumbs-up, especially for Java projects, where it adds a powerful defect checker.

Unfortunately, the online version of the review does not include a screenshot, which is fairly important for seeing what the product is like. So, here is the starting page, from which you can drill down into CQ2's other dashboard windows. As you can see, it's a clean, uncluttered interface.

More on Writing Your Own Language

Martin Fowler has a great series of articles on writing your own language. The advice is primarily oriented towards little languages--or what are now called domain-specific languages (DSLs). However, much of the information applies to more ambitious language projects as well.

Writing your own language -- How to choose a VM

Let's say you want to write your own language, possibly even your own domain-specific language (DSL) and you want to run it on a VM. Which VM do you target?

You might instinctively think of the JVM (especially because there are so many tools to help you target the JVM) or the .NET CLR. These choices have the benefits of being highly optimized platforms on which numerous, substantial libraries already run. However in many cases, they will not be the best choice. For several reasons: they're big and difficult for users who are not technical to install (and you can't bundle them with your app); they can't be embedded into other applications; they might not support features you need.

There are many interesting alternatives that are small, reasonably fast, and have active communities supporting them.

There are, of course, the Perl and Python VMs. As you'll quickly see upon careful examination, however, both of those VMs are intimately tied to the languages they run. (No suprise there.) In addition, neither VM was really designed for targeting by other apps, so info on developing languages for them is not widely/conveniently available.

Other VMs do provide extensive support for new language developers, because they know this is the only way for them to build community. Here are three among the many you could choose from:

Lua, which consistently is the fastest performing VM outside of the JVM. It's also small, open source, widely used, and easy to embed. It's also very well documented and supported by an active community.

Neko VM, small, easy to embed, very actively supported by its developer. Particularly amenable to embedding in C applications.

Pawn, designed primarily for embedding. Its own default language is a lot like C.

These VMs are all open source. Other VM candidates are listed here, although this list is far from complete.

Most of these VMs encourage your compiler to output not bytecodes but source code using their native language. In most cases, this is the right approach because 1) the VMs can optimize your code better than you can write in their bytecode format, and 2) it saves you a whole lot of aggravation by not having to learn the bytecode and the minutia of VM internals. (Some knowldege of VM interals will, of course, be necessary.)

That being said, you do you decide which VM fits your needs best? Some principal considerations include (more in the link below):

• Does the VM have native support for the data items you need?
• Does the VM support the language features you need (garbage collection, multithreading, tail recursion, etc.)?
• Does the VM support the performance features you need (optimization, JIT compiler, etc.)?

Once you have found a virtual machine that suits your needs, you have only to check out the community behind it, to make sure you're not nearly or completely on your own.

Note: Lambda the Ultimate is a terrific site for aficionados of programming language development. Here is a link to a post on this topic that might shed further light. If this area interest you, definitely tag/bookmark the Lambda site.

Final note: JetBrains, the makers of the popular IntelliJ IDE, have a mechanism that can be modified to provide an IDE extension for your language.

Dijkstra's Three Rules for Project Selection

Edsger Dijkstra promulgated these rules in the context of scientific research, but I think they work well for selection of programming projects or, really, most intellectual endeavors that might break new ground.

The Three Golden Rules for Successful Scientific Research.

This note is devoted to three rules, the following of which is necessary if you want to be successful in scientific research. (If you manage to follow them, they will prove close to sufficient, but that is another story.) They are recorded for the benefit of those who would like to be successful in their scientific research, but fail to be so because, being unaware of these rules, they violate them. In order to avoid any misunderstanding I would like to stress, right in its first paragraph, that this note is purely pragmatic: no moral judgments are implied, and it is completely up to you to decide whether you wish to regard trying to be successful in scientific research as a noble goal in life or not. I even leave you the option of not making that decision at all.

The first rule is an "internal" one: it has nothing to do with your relation with others, it concerns you yourself in isolation. It is as follows:

"Raise your quality standards as high as you can live with, avoid wasting your time on routine problems, and always try to work as closely as possible at the boundary of your abilities. Do this, because it is the only way of discovering how that boundary should be moved forward."

This rule tells us that the obviously possible should be shunned as well as the obviously impossible: the first would not be instructive, the second would be hopeless, and both in their own way are barren.

The second rule is an "external" one: it deals with the relation between "the scientific world" and "the real world". It is as follows:

"We all like our work to be socially relevant and scientifically sound. If we can find a topic satisfying both desires, we are lucky; if the two targets are in conflict with each other, let the requirement of scientific soundness prevail."

The reason for this rule is obvious. If you do a piece of "perfect" work in which no one is interested, no harm is done, on the contrary: at least something "perfect"—be it irrelevant—has been added to our culture. If, however, you offer a shaky, would-be solution to an urgent problem, you do indeed harm to the world which, in view of the urgency of the problem, will only be too willing to apply your ineffective remedy. It is no wonder that charlatanry always flourishes in connection with incurable diseases. (Our second rule is traditionally violated by social sciences to such an extent that one can now question if they deserve the name "sciences" at all.)

The third rule is on the scale "internal/external" somewhere in between: it deals with the relation between you and your scientific colleagues. it is as follows:

"Never tackle a problem of which you can be pretty sure that (now or in the near future) it will be tackled by others who are, in relation to that problem, at least as competent and well-equipped as you."

Again the reason is obvious. If others will come up with as good a solution as you could obtain, the world doesn't loose a thing if you leave the problem alone. A corollary of the third rule is that one should never compete with one's colleagues. If you are pretty sure that in a certain area you will do a better job than anyone else, please do it in complete devotion, but when in doubt, abstain. The third rule ensures that your contributions --if any!-- will be unique.

***

I have checked the Three Golden Rules with a number of my colleagues from very different parts of the world, living and working under very different circumstances. They all agreed. And were not shocked either. The rules may strike you as a bit cruel... If so, they should, for the sooner you have discovered that the scientific world is not a soft place but--like most other worlds, for that matter--a fairly ruthless one, the better. My blessings are with you.

10 People Who Don't Matter in Tech

Interesting choices (Linus Torvalds, Jonathan Schwartz, Steve Ballmer--wow) made by Business 2.0. After reading their reasons, I can't say I disagree entirely with their selections. link

VMware Workstation 5.5 Impresses

For the last year or so, I have been using virtual machines (not the Java/CLR type, but the ones that enable you to run a guest operating system on a host machine) for two purposes: software reviews and long Web searches.

The use in reviews saves from me from loading and unloading software and slowly corrupting my Windows registry and dotting my win32 directory with leftover DLLs. The use in long Web surfs is to protect myself from malware. In the latter case, if your virtual machine is corrupted by malware, the entire damage is contained to the small set of files that constitute that virtual machine. You blow them away, clone an existing virtual machine, and you’re back in business. Nothing else is affected–nothing on the host system, nor any other virtual machine. Pretty cool.

Anyway, for the last year I have been using Virtual PC from Microsoft, which does a creditable job. This month, I switched to VMware Workstation 5.5. What a difference! The most conspicuous advantage of VMware is performance. When you’re running VMware at full screen, it’s hard to tell that you’re in anything but your regular, native desktop. I haven’t benchmarked it, but it’s very, very close. In contrast, with Microsoft Virtual PC 2004, you always know you’re in a virtual machine because performance is very slow. Opening dialog boxes, moving windows, starting up applications–all these actions arevisibly slow. So, not only does everything take a long time, but it is harder to assess the performance characteristics of software.

So a recommendation and a tip of the hat to VMware for a great implementation.

BEA’s AquaLogic: What is it?

I met with folks from BEA earlier this week about their recently released WebLogic 9.0 application server and the AquaLogic software the company has been promoting heavily. AquaLogic, I now understand, is the company’s entry in the ESB space. It includes the ESB per se and registry functions. The ESB is based on the company’s former JMS messaging middleware, which it developed in-house. I expect I’ll be reviewing the products at some point later this year.

[Update (March 2006): I did end up writing a review of AquaLogic for InfoWorld.]

Search Engine Page Depth

In my post regarding Google’s Desktop Search Engine, I mentioned several objections I had to the technology. One of them was that it did not search past 5,000 words in documents.

Subsequent research has brought to light that this partial-search is a common trait of search engines (although most of them go much farther than 5,000 words). How far down the document the indexer will go is a metric known as page depth. And, according to searchenginewatch.com, there is considerable variety in the page depth. According to the site’s 2003 figures (the latest one they have), here is how the major search engines measure up in terms of page depth:

• Google: 101KB
• MSN: 150 KB
• Yahoo: 500 KB

So, as you see: In terms of results, a search engine only gets you partial information on what’s known to be available. Such that the absence of a result should not imply that the item does not exist. In fact, the item might even be sitting in the search engine’s document cache. However, it is simply not in the engine’s index.

Rogue Wave C++ Library now Open Source

Rogue Wave (now a division of Quovadix) has open-sourced its implementation of the standard C++ library. The company gave the library to the Apache Software Foundation, which stuck it immediately in its incubator system, from which it presumably will emerge months from now. In its hey day, Rogue Wave was considered a vendor of some of the best-written libraries in C++, so I suspect quality on this will be pretty good as well. The link for more info is here.

Portable Libraries: Slim choices

As I move forward with my book on Intel EM64T 64-bit extensions to x86, I want to present some complete programs. I want the code to be portable, and I don’t want to write every routine and data structure by hand. What I need is a portable library for C that can be used for free by readers. C/C++ developers with the same problem have a few choices:

For GUIs:

• Qt from Trolltech
• GTK

For everything else:

• APR: Apache Portable Runtime (C)
• NSPR: Netscape Portable Runtime (C)
• ACE (C++)

Most of what I need falls in the domain of one of the last three libraries. There are problems with each choice, however. The APR is the most actively supported of these, but it has one horrible failing: zero documentation. And because it uses a very specific model, if you don’t grok how it comes together, you will find it impossible to use.

NSPR is well documented, but is currently supported by only one person. The limitation of his time and his resources make it difficult to commit to the library with heart and soul. Although, of the three general-purpose libraries, this is probably the one that appeals to me the most.

The ACE library is another low-documentation product. In fact, other than Doxygen lists of API calls, the only documentation extant is a book. It seems to me that if you want the lib to be used a lot, the authors should make parts of the contents freely available, enough to get potential up to speed. If the library and the book are good, most every user will buy a copy. But I won’t commit to either one without some certainty going in that I will arrive satisfactorily where I want to go.

So, what to do? I will use the Qt library, which is actually much more than a GUI library. It provides support for threads, networking, sockets, database access, and XML–which are the features I need. The library is dual-licensed: free for GPL’d projects and licensed for a fee on all other projects. Since my code will be open source the free Qt solution works. Plus it’s a lovely crafted library. With great docs–not to say coverage via several books.

The oddest thing in this whole search is the APR’s lack of documentation. Despite having an active list of contributors, nobody has done docs. As a result, the only developers who use it are those who wrote or worked on the library. Seems like a stupid obstacle. I’d help out on docs if I had anyway at all to figure out how the library works, but there is literally no info.

Streaming Desktop Major League Baseball: Not Ready Yet

As this year’s baseball season started up, I decided to test Major League Baseball’s new option for watching my favorite teams: streaming video of their broadcasts. A subscription costs only $14.95/month and I can watch the games of as many teams as I want. At first, this was fun, but as time passed the limitations of the subscription have led me to consider cancelling it. The problems are technological and political.

The political problem is that Major League Baseball (MLB) has exclusive national TV contracts with various networks. “Exclusive” is the operative word. For example, due to these contracts there are no streaming video of baseball games on Saturday. MLB also has regional contracts, in which a network might broadcast a game I want to watch. Even if the network doesn’t broadcast it in my home market, the mere fact that they might have precludes me from seeing it on streaming video. So, in essence, I have a 6-day/week subscription with numerous holes in the remaining schedule.

The technology problem is the video itself. It’s delivered in a 2″x3″ screen in the browser. You can make the image larger, but you lose picture quality as you expand it. The video also has a problem that it frequently stutters and then blanks out all together. In the latter case, the screen just shows the MLB logo. You have to restart the transmission to get the picture back. This happens more in the Firefox browser than IE, but it definitely happens frequently (about once per inning) in IE as well.

The bottom line is that between the technology glitches and the politics, streaming MLB video is a frustrating experience.

[Update (April 2006): I subscribed again this year and had to cancel. The service has not improved in the intervening time.]

Google Desktop Search: good, but limited

Google’s enterprise desktop search (available here at no cost) is a useful tool. It does pretty much what you’d expect: it reads through your docs and e-mail and gives you an instant search capability. It has saved me several long searches for docs I knew were lying around but I couldn’t remember exactly where.

For all its benefits, however, the technology does have some distinct limitations:

it only searches the first 5,000 words of any document

you cannot remove or add items to the search database. (You can prevent it from looking in certain directories, but once its sucked up a document, you can’t remove it from its database.) The software crawls how and when it wants to. If you want it to perform a new crawl, you have only one option: uninstall it and reinstall it. No kidding!

There’s no concept of security. You cannot password-protect access to the search engine.

The engine does not search Eudora mailboxes.

The display does not highlight the searched-for keyword in the document.

For these reasons, true enterprise search tools such as IBM’s Information Integrator OmniFind do not have much to fear yet from this free offering. “Yet” is the operative word. Obviously, all these limitations are intentional, as Google has demonstrated it has the technology to get around them in its other products. So, at any time it could fold these features into its free offering.

If you can live with these limitations, you’ll definitely find the search engine useful.

Preparing for JavaOne 2005

This year’s JavaOne conference is around the corner. This guide tells you how to prepare properly. Enjoy!

ISVs: Use Embedded Databases, Please

This week, I completed my review of CodeAssure, a static source checker that focuses on security problems. It ships from Virginia-based Secure Software Inc. The product analyzes a C or Java codebase and generates a file of project vulnerabilities each time it’s run. This data is then shown to developers to help them locate possible security weaknesses. A separate manager’s console can compare these files, quantify project progress, and display this in a dashboard for project management purposes.

For this comparison, CodeAssure requires and enterprise DBMS (be it Oracle, DB2, or SQL Server) to store these data runs. (Up to the customer to cover the cost of obtaining the license for this.) Primarily because of this DBMS, Secure Software flies out an engineer to set up and install the package each time a customer buys a copy of CodeAssure. The question is: Why not make this database embedded? If the company used an embedded database, it could:

ship the product as a turnkey product

save the cost of sending out an installation engineer

lower their customers’ implementation costs

Those are compelling benefits.

There are some great embedded databases out there. Sleepycat software’s Berkeley DB being probably the best known and arguably the best implemented. But it’s certainly not the only choice. There are many others, some of which are completely free for commercial use, such as Apache Derby.

The reason for not using them, I believe, is that ISVs simply don’t think in terms of this kind of solution. CodeAssure is the defining example of a product where embedded databases should be used: it’s an applaction that is the sole consumer of the data it generates. I asked the Secure Software why it had not chosen the embedded model and the staff responded that they generate so much data, it could be handled only by an enterprise DBMS. This suggested to me that they had not tested embedded products sufficiently.

Embedded databases are robust and highly scalable. A case in point is Sleepycat’s product, which is thefront-end cache for Amazon.com’s product/inventory database. By size, that database is among the top 20 commercial databases in the world.

In the hybrid enterprise/embedded space, there are products like MySQL, which to my knowledge have not exhibited scalability issues.

Secure Software did allow that it was beginning to examine embedded options for future releases. I encourage them to do so, as I do all ISVs who cannot articulate a specific reason that an embedded solution is insufficient for the need. This design will make installations and deployments easier, plus save customers the cost of paying for new and expensive DBMS licenses. And so, we’ll all be a bit happier.

Choosing a programming language

Programming languages rise and fall in popularity. This fact places pressure on managers to choose the right language when starting in on a big project. In my case, it places pressure on authors when choosing the language with which to exprese a concept in code. I have always chosen C, reasoning that C is readable and understandable by all C++, C#, and Java developers–which is about as broad a base of readers as I’m ever likely to reach. (Knuth’s discusses this difficult choice in the just-released Fascicle 1 of the Revision to Volume 1 of “The Art of Computer Programming” from Addison-Wesley, ISBN 0-201-85392-2)

An interesting website list the interest in specific languages based on counting Web searches for topics relating to the language. Results are normalized to avoid spurious one-off results. The methodology is explained in detail on the website. The results for the last few years are shown. I am surprised that C is still #1 (actually, it reclaimed the prize from Java a few months ago.) What is surprising is that C and Java lead C++ by such a wide margin. If I’d been asked to guess, I would have swapped the positions of C and C++ and/or possibly placed Java in first place.

There are two other interesting items. Despite rumors to the contrary, COBOL is still thriving. It’s in the most active class (those assigned a grade of ‘A’) and a fair bit ahead of Fortran and Pascal. Also, the conventional wisdom about the ‘P’ scripting languages (the three candidates for the letter P in the LAMP stack) appears to be correct. In descending order of interest, they are: Perl, PHP, and Python. Which I suspect is what most people would have guessed.

Anyway, use this info as a data point for your own decision making and in your choice of language skills.

Qt 4.0 from Trolltech

I met with the folks from Trolltech last week for a presentation on Qt 4.0, which ships at quarter end. For readers unfamiliar with Qt, it is a highly portable toolkit (that is, GUI, collections, database, threading, XML functions and others) that’s used by many ISVs. It is dual-licensed, meaning that if you use it on a free, open-source project, you can get it at no cost. For anything else, you’ve got to buy a license from Trolltech. The free version is the basis for Linux’s KDE (the K Desktop Environment).

I worked with Qt 3.x for well over a year and I was very impressed. Qt is one of the cleanest libraries available. It is written C++ and is well thought out, well designed, and after a little experience it becomes intuitive to use. On Windows, I think it’s a better interface to the operating system than Win32 is. Not only because it’s better designed (You soon know the APIs without looking them up because they’re so consitently named.), but because it’s portable (to Linux, Mac, UNIX). Version 4.0 is a major upgrade. The Trolltech folks told me that they added numerous functions and also redesigned the painter so it renders far more-advanced graphics far faster. And, indeed, the demo blasted the screen.

Qt 4 also adds considerable threading support, which I think will be an important, even if underappreciated, benefit. When I was co-authoring the book on Hyper-Threading Technology for Intel Press, I had to write the same program twice. Once for Windows threads and once for Pthreads (the Linux/UNIX threading model). I really wanted a portable way to write the code. If Intel Press had allowed it, I could have used Qt 3 (I didn’t even ask because prior to Qt 4, Windows versions of the library required a paid license–there was no freebie for open-source projects.), but Qt 4 with its greater threading coverage and friendlier Windows license would be a slam-dunk today.

Qt 4 looks like an important release. (I suspect I’ll be reviewing it once it ships.) And with Macs soon migrating to Intel platforms, some code will need to be rewritten. Qt 4 probably represents a better solution than porting and tweaking old Mac code, especially for Mac developers who are entertaining any ideas of moving to Windows later.