Although I am a believer in free markets and the benefits of competition, industry has a responsibility to work together on the foundational layers to build security, quality and reliability from the ground up to advance the professionalism of the field. In essence, the information technology industry must emulate other engineering disciplines, or technological disasters and cybersecurity holes will worsen.

Daconta is uneasy with the number of platforms and methods available to software developers, and sees ever-more options and disruptions in the near future; IPv6 and 64-bit computing seem to trouble him particularly. We’re already balkanized and disorganized, how can we possibly expect to produce reliable and useful software with all this messy innovation happening?

The answer, of course, is control. Lots of it. Specifically, three proposals:

• Licenses for software developers
• A new, reliable, layered software platform developed by the NSF and DARPA
• Treat software like engineering, not art.

Gracious. I barely know where to start. Let’s try to imagine the software development world in five years, with these proposals in place.

Software development is now a licensed activity. Like an architect or a mechanical engineer, you have to pass an exam and perhaps post a bond to practice the discipline. There’s probably a professional association, like the America Bar Association or the American Medical Association, to administer the credentials.

This licensing regime is actually a pretty good idea, because all software has to be developed according to some very specific methods, with plenty of testing and documentation to back it up. So rather than letting any fool with a compiler write software, they’ll have to spend a year or two learning the right way to write code. The process is cumbersome, but every piece of software that gets compiled is perfect. At least, as perfect as we know how.

The licensing and formal methods are only possible, of course, because we have a government-directed platform that we must build on. Anyone who wants to run software in the government must do so on this stack.

It sounds as though Daconta would like a broad mandate for NASA-style code development. You can probably see where I’m going with this. Another way to tell the story might be:

There is now a government-owned platform that every government program is mandated to use, from clouds to mobile phones. Any software built on that platform must submit to rigorous, independent testing before it is deployed. Imagine ISO 9000 and Common Criteria having a baby with teeth. Anyone writing software must be licensed to do so in the United States. As a result, the pool of available programming talent is decimated. The costs of developing software for government rise, naturally.

The pool is further diminshed because developers who want to work with the latest hardware or software no longer work for agencies or contractors — they’ve wandered back to the private sector, where they can enjoy the fruits of the free market. The government software platform quickly begins to show its age, since the only developers on the platform are those that are paid to use it. Platforms that people truly want to use are in the open market, innovating at their leisure, out of the reach of government agencies.

As the government is no longer able to consume most commercially available software, it is now back in the business of writing software itself. More accurately, it is back in the business of hiring system integrators to write that software on its behalf. It’s the 1970s all over again. Budgets explode, and innovation grinds to a halt. It’s all the agencies can do just to tread water. System integrators, of course, are delighted. They’re now commanding outrageous salaries for the few programmers trained and willing to work on this mandated government platform.

Let’s hope there’s a better way.

This doesn’t diminish the problem that Daconta is hinting at, of course. Software reliability is certainly something to worry about. But there is no single solution or set of policy prescriptions that will solve the problem. I don’t think that imposing additional controls on the development of software makes sense, certainly not for all cases. There are already robust certification regimes and methods for software that does very important work: fly an airplane, control a nuclear reactor, and so forth. We don’t need that kind of scrutiny on my game console or desktop.

It’s important to note that these robust certifications only evaluate the software itself, not the people who make it. This is what’s great about software: we can examine the final product before it’s distributed. I don’t really mind if my software is written by a clever 7 year old. If it’s doing the job it’s supposed to, that’s fine with me.

This focus on ends, rather than the means, is something you can’t do with a building or an airplane. With software, we can change our minds with far fewer consequences. We can thoroughly test and scrutinize before it’s in a customer’s hands. When we do find a flaw, it’s easier to patch software than it is a 777 or a skyscraper. We should take advantage of that fact, not try to make software rigid and inflexible just because we know how to manage rigid and inflexible things. Because software has these unique properties, we have the freedom to bring more or less scrutiny to bear, depending on the circumstances. Which is what we’re doing already, through programs like the federally-funded Software Engineering Institute at Carnegie Mellon.

So where Daconta sees mayhem and chaos, I see creativity and innovation. There are many opportunities to improve the reliability of our software, but none of them have to do with the process by which we arrive at a particular piece of code. Projects like David Wheeler’s OpenProofs, for example,  can provide the tools we need to be mathematically sure software is doing what we intended. The Linux Test Project does this for Linux, and it was inspired in no small part by governments’ Common Criteria mandates.

This is, in fact, how the process should work. The government should set the requirements for reliability and assurance, and allow the private sector to innovate its way toward those requirements. If we only create software that we can understand perfectly, we lose the ability to be creative, to innovate, and to take advantage of the collective intelligence and cleverness of millions of software developers. We will never eliminate risk in software, but we can manage that risk, not through more stringent controls, but by encouraging as many smart people as possible to address the problem.

[This was also published at opensource.com and GovFresh]

Open standards also keep costs low for buyers, who have many options and a minimum of friction when they want to switch from one implementation to another. Because the standard is open, there is no danger of being locked into a single vendor since anyone can create a new implementation against the standard. Since open standards will always exist, there’s no danger of the standard disappearing, becoming unsupported, or being later made proprietary. An open standard will encourage these efficient, durable markets for as long as the standard is useful.

Too often, plain-vanilla standards are conflated with open standards. It’s a shame, because it’s a very important distinction. Anyone can create a standard and collect a large ecosystem around it. Microsoft Word’s document format is a great example of this. It is a standard only inasmuch as it is ubiquitous. In fact, it was only documented formally in February 2008, before which anyone who wanted to use the format had to guess at its inner workings or pay a fee to Microsoft. So while the Word document format was a standard, albeit ad hoc, it created no market, discouraged competition, and represented a serious barrier to entry for anyone who wished to create a word processor that would compete with Microsoft Word.

By contrast, the Open Document Format was developed collaboratively and is freely available to anyone who wishes to implement it. The demand for a viable alternative to the Microsoft Office document formats had found an outlet, and there are now dozens of applications which support the format, including Open Office, Google Docs, and even Microsoft Word itself.

So just because something is a standard, that doesn’t mean it’s open.

What’s Different About Open Standards

In order for a standard to be open, it has some very specific characteristics. Not surprisingly, there are many opinions about what those characteristics are, and Wikipedia has an excellent collection of these. Though there are dozens of definitions available, there is broad agreement that users of an open standard must have a few specific characteristics, which I’ve described as freedoms:

• The freedom to read and implement the standard at no cost.
• The freedom from patent encumberances, either through royalty-free licenses or a promise of non-assertion by the patent holder.
• The freedom to examine and participate in the development of the standard, through a consensus or majority decision-making process.

There are many other characteristics described by these various governments and international bodies, many of which are less definitional than they descriptive of a healthy and useful open standard. We will return to those shortly.

You could say that the freedom to obtain and implement a standard at no cost is the defining characteristic of an open standard. There is no fee, no contract, and everyone is using the same set of instructions. This freedom, though, only distinguishes open standards from closed, for-fee standards. Standards which are proprietary but available at no cost, like the Microsoft Office document formats, cannot be called open. This is at the heart of the confusion over open standards – many do not understand that truly open standards provide more freedoms than no-cost standards.

Freedom from patent encumbrances is crucial to making the first freedom practical. A prospective implementer needs to be assured that they won’t subject themselves to rent-seeking to gain compliance.

The ability to participate in the ongoing development and refinement of a standard is the most unique characteristic of open standards. It is how a standard stays open and viable over time. Open standards must have a facility for revision and correction that accommodates its community of implementers and users. If there is no facility for this, the standard is simply a mandate from one vendor or industry to others. This introduces the possibility of features that favor one implementation over another. Worse, a sudden and unexpected change in the standard could thrust existing implementations into non-compliance – essentially casting all existing implementations out of the ecosystem, which requires time, money, and effort to re-enter. This uncertainty decreases the likelihood that the standard will be useful and thus widely adopted, both now and over time. An open process removes this uncertainty and significantly improves the usefulness of the standard.

So describing a standard as open means much more than “no charge”. If a standard is meant to be a neutral set of rules by which implementations compete for users, simply being free (though useful) is insufficient. An open standard cannot be controlled by a single party, even if it is available at no cost. Without a transparent governance, the standard is simply free.

The Role of Open Source Software

I’ve framed these characteristics of open standards as freedoms to invite comparison with the four freedoms of open source software:

• The freedom to run the program, for any purpose (freedom 0).
• The freedom to study how the program works, and change it to make it do what you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to improve the program, and release your improvements (and modified versions in general) to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.

Open source and open standards are the same process, applied to two different problems: open source is for computer code, and open standards are for data. The details vary, but when we talk about the principles of open standards or the benefits of open source, we’re describing the same collaborative innovation process.

Returning to the list of open standards definitions at Wikipedia, you’ll see that a few of these definitions encourage or mandate multiple implementations of the standard. This is wildly important, but I don’t think it’s definitional for open standards. Multiple implementations ensure that the standard is practical, which is not something to take for granted when complex technical standards are developed by committee. Having multiple implementations also means choice for the user, which is central to the value of a standard. So multiple implementations are great.

Now imagine if at least one implementation of this open standard is done with open source software. We not only have a freely available standard that anyone may use, but a freely available implementation of that standard that can be examined and re-used by other implementers. An open source implementation is also an “implementation of last resort” should every implementation disappear or go out of business. Those who rely on the standard have not lost their ability to interoperate. An open source implementation inoculates an open standard against the future, and ensures that there is always at least one competitor in the market. This is good for the standard, and good for the users.

Contrariwise, imagine an open standard with only proprietary implementations. This is scenario works, but substantially undermines the stated goals of an open standard: choice, freedom, and durability.

We may have many proprietary options before us, but without an open source implementation, it is significantly more difficult to create an alternative. You’re starting from scratch each time, so the bar is unnecessarily high for newcomers to the market. Likewise, our freedom to change implementations with a minimum of friction is diminished.

Ideally, every implementation would provide some assurance of interoperability that facilitates my ability to freely choose among implementations. This is not always the case, though, and it’s possible for vendors to “embrace and extend” a standard, which is code for creating an implementation that adds vendor-specific hooks which may interfere with my ability to switch to another implementation. If all the implementations are proprietary, they could all be compromised in this way, leaving me with no true implementation of my otherwise open standard.

If I have no open source implementation, I am reliant on the continued success of the proprietary implementations. If a company goes out of business or leaves the market, I may be left with no viable options. Of course, other competitors could appear – I could even write an implementation myself, though that’s often so expensive as to be impractical. This is all unnecessary if there is an open source implementation available. Again, it is the implementation of last resort.

So while I think it’s excessive zeal to say that an open standard must have an open source implementation, it’s wise to examine claims about “openness”. If there is an open standard that lacks a transparent governance, and has no open source implementation, you have to seriously consider the long-term viability of that standard, and its ability to provide a properly functioning market.