I noticed that Brian repeated the misunderstanding that San Francisco had the nation’s first open source policy. I don’t want to diminish his larger argument, but it’s important that we give credit where credit’s due. So for the record:

• July 1, 2004: OMB issues OMB-04-16, making clear that open source can be used in the Federal Government
• September 30 2009: Portland, OR is the first city to issue an open source policy.
• October 16, 2009: The US Department of Defense CIO issues a memo reiterating that open source software is commercial software for procurement purposes, and encouraging DOD branches to include open source when they’re picking software.
• January 7, 2010: California‘s open source policy is published.
• February 1, 2010: San Francisco, CA issues their open source policy.

These are just what I could find, of course. If you know of others, let me know! If you’d like to see a comprehensive history of open source battles in national and state governments around the world, CSIS maintains an annual survey intuitively titled “Government Open Source Policies“. Even just skimming it, you’ll be surprised at how little progress the United States has made in open source policymaking.

I’m not sure I can recommend reading all of the plans cover to cover, but if you’re an advocate or have a vested interest in the future of a Federal agency, these plans are fascinating peek into each agency’s interior life. It’s not just the content of the plans, which run from exciting to comical to mundane. You can also learn a great deal about how agencies view themselves from the way these plans are presented and marketed. It will come as no surprise that the Department of Justice’s rather unlovely document spends a lot of time thinking about reducing its FOIA backlog. The Department of Energy clearly understands itself to be a first a research organization, based on its flagship data sets. The Department of Defense plan is crisp, to the point, and focuses on getting the behemoth to better collaborate and interact with other agencies, rather than the public.

The organizational psychology betrayed by these plans is for another post. My interest is in where agencies found open source. I’ve long advocated for open source as a concrete, tangible way to encourage collaboration between agencies and between the government and its citizens. I was pleasantly surprised, frankly, to see how many agencies agree. Here’s what I found, in no particular order.

US Agency for International Development

The USAID plan was a total surprise. I had no idea how many open source initiatives were being conducted by USAID. Page 30 contains this gem on their Global Development Commons work:

With over four billion subscribers in the world, the mobile phone is often the key to connecting and exchanging information with people in developing countries. The 2008 USAID Development 2.0 Challenge, implemented by the Global Development Commons, invited innovators and entrepreneurs from around the world to participate in a global competition to seek access to information and build new connections to the global community. Crowdsourcing and Open Innovation have become increasingly important engines of innovation globally, leveraged by the commercial, non-profit, academic and government sectors to identify opportunities and solve problems. USAID’s Development 2.0 Challenge yielded 115 submissions using high impact, low-cost, open source solutions.

The winner among the 115 submissions was the RapidSMS Child Malnutrition Surveillance system, which “enables health practitioners to share and track children’s nutritional information with the touch of a cell phone.”

The agency also operates the Intra-Health OPEN Initiative, which is “a suite of free open source solutions to supply health sector leaders and managers with a collection of new tools.”

Social Security Administration

The Social Security Administration is another open source underdog. Imagine all the pent-up innovation they can unlock once this project is underway:

We are in the process of creating internal capacity to host websites and applications based on open-source software solutions and we look forward to a lively exchange of ideas and program code within the growing Federal open–source software development community;

As part of SSA’s fifth goal, “making government more sustainable”, they see open source software as an essential tool:

We are a Federal leader in the use of Health Information Technology. Our work with the private sector may yield transferable ideas and tools. We will share our results and products as appropriate. For example:

• We look forward to sharing the products of our open–source platform efforts across the growing Federal open–source development community, as well as partnering with other agencies in future endeavors; and
• We are in the process of designing and developing an Electronic Technology Repository for communities of innovation. We expect this repository to employ open–source social networking and other tools to permit users to better manage agency knowledge, avoid unproductive duplication of effort, and share experiences. The repository will support the storage of shared materials and project artifacts, discussion boards, wikis, blogs, subscription feeds, and other pertinent information. We envision sharing these resources with other Federal organizations as well.

Others have criticized open source as being irrelevant to the open government movement, but I think interagency collaboration doesn’t happen anywhere near as often as it should, it can be made easier with open source, and it’s outstanding that the SSA seems to agree.

Department of Defense

The DOD has been using open source software for years. Though I was a bit surprised that it wasn’t explicitly mentioned in the DOD’s open government plan, and even more surprised that the much-hyped forge.mil project isn’t referenced at all, the plan includes a number of initiatives that happen to take advantage of open source tools:

The Wikified Army Field Guide, based on Mediawiki, will allow warfighters to collaboratively edit the Army’s field manuals, allowing the documents to be more accurate and responsive:

…as the battlefield changes rapidly, field manuals must keep pace.  Under the traditional process – in which a select few were charged with drafting and updating field manuals –manuals often failed to reflect the latest knowledge of Soldiers on the ground.

Using the same free software behind Wikipedia, the Army’s “wikified” field manuals invite military personnel – from private to general – to collaboratively update the Army Tactics, Techniques and Procedures Manuals in real time.  In so doing, the Army provides a secure means for battle-tested Soldiers to share their experience and advice from the field.  Wikified Army Field Manuals ensure the men and women who serve our Nation have access to the best possible information when they need it.

This is a very exciting opportunity to capture all the innovation happening “at the edge” and quickly incorporate it into useful, official documentation. It makes so much sense, I’m surprised it hasn’t already been done.

The plan also highlights XMPP, which is a tremendously popular instant-messaging protocol that runs, among others, Google Talk. It may surprise you to learn just how much XMPP’s most popular implementation, Jabber, is already being used inside the DOD. The Defense Connect Online program uses Jabber to provide secure IMs inside the DOD, and they announced in November that this would be opened up to the outside world. Because they standardized on an open standard with robust open source implementations, literally dozens of different chat clients are now available to these non-DOD DCO users.

It’s interesting how both Jabber and the Wiki Field Manual projects aim to improve collaboration, and do so on highly collaborative open source platforms. I don’t think that’s an accident.

Department of Homeland Security

VirtualUSA is DHS’ flagship initiative, which couldn’t be more appropriate. From page 23 of the DHS Open Government plan:

On December 8, 2009, Secretary Janet Napolitano publicly launched Virtual USA (vUSA), an innovative information-sharing initiative that draws on practitioner input to help Federal, State, local and Tribal first responders collaborate to make fast, well-informed decisions. vUSA integrates existing frameworks and investments to provide real-time access to operational information—such as weather conditions; traffic; the location and operational status of critical infrastructure; fuel supplies; availability of emergency shelters and medical facilities; and other critical information—that allows users to improve situational awareness and to respond quickly in emergencies.

vUSA currently operates as two pilots – one in eight southeastern states: Alabama, Georgia, Florida, Louisiana, Mississippi, Texas, Virginia and Tennessee; and the other in five states in the northwest: Alaska, Montana, Idaho, Oregon, and Washington. In Virginia alone, vUSA reduced response times to incidents involving hazardous materials by 70 percent.

VirtualUSA is much more revolutionary than this lets on. It’s a very disruptive piece of software in its space. FCW has a succinct overview of the project’s history and what it means for first responders. DHS is funding this project, based on open source and open standards, in part because it wants to encourage collaborative toolbuilding and cooperation among the states, and also because this capability is too important to be in the hands of a single GIS provider, like Google or ESRI. Because it is an open source project, and uses open standards, VirtualUSA ensures that critical assets are not locked into a single vendor, and simultaneously lower the barrier to entry for new GIS vendors.

Department of Commerce

On page 18 of the Department of Commerce plan, under “Open Source Information Technology”, we find some familiar prose:

Also emerging from Commerce’s Open Government Ideascale community was a suggestion to “become more open through the increased use of open source software.” The Department has already begun using the open source tool, Drupal, for a number of its new websites and plans to increase this use in the future. Using open source technology will allow Commerce to develop new technologies and collaborate more readily with the public and other government agencies, and within the Department itself.

To make this happen, the Office of the Chief Information Officer and the Office of Acquisition Management will be consulted to ensure that open source offerings are fully considered during procurement processes. That consideration will include the value that the Department can receive through increased collaboration with the public and as a contributor to open source communities.

Nothing short of victory at Commerce for Open Source of America, whose suggestion this was. Congratulations!

Department of Labor

You’ll find open source in the strangest places. Until I read Labor’s plan, I didn’t appreciate how much data the Department of Labor is responsible for analyzing and disseminating. With that in mind, it makes perfect sense to find this initiative on page 29:

Create a “Developer’s Corner”

We plan to establish a “Developer Corner” on www.dol.gov/open that specifically targets and engages developers. Our goal is to make it as easy as possible for developers to re-purpose our data, provide feedback, get technical help, bring developers with similar interests together and, ultimately inspire the best possible uses of our data for the benefit of the public. Ideas under consideration include a bug tracking system, RSS feeds for dataset changes, dataset versioning, public code competitions, data authentication, and an ideation platform to prioritize developer needs.

I think every department and agency that distributes data to the public (which is to say, all of them) should follow Labor’s lead and establish their own Developer sites. There’s no better way to stay engaged with this very powerful community.

Health and Human Services

The outstanding NHIN CONNECT project, which has a thriving open source community, got a mention on page 56 of HHS’s plan:

Nationwide Health Information Network – Direct

A key component of the Nation’s emerging health information technology infrastructure is the Nationwide Health Information Network (NHIN) – a set of standards, policies, and services that enable the secure exchange of health information over the Internet. “NHIN Direct” is the latest development in the evolution of the NHIN. It’s an important effort to develop a “lightweight on-ramp” to the NHIN that will enable simple, direct exchanges of information between providers, labs, pharmacies, and consumers — and which will be easy to adopt and implement. In a process that launched on March 1, NHIN Direct is being designed in close collaboration with the community of potential users, with the entire process taking place in the open, in public, on a NHIN Direct wikispace. NHIN Direct will then be implemented in real-world tests and deployments by members of the community – with HHS’s Office of the National Coordinator for Health Information Technology (ONC) playing a coordinating and convening role. The gist of the NHIN Direct strategy is to utilize a community-driven approach to ramp up and power NHIN Direct-powered health information exchange.

I should mention that my employer, Red Hat, is involved in this project.

National Science Foundation

There was nothing explicitly about open source in the NSF’s open government plan, but their plan is worth mentioning anyway, as the NSF already does a tremendous amount of work in the open source community. Here’s a search for “open source” on their web site, which yielded over 5,000 hits when I last tried it:

http://search.nsf.gov/search?access=p&output=xml_no_dtd&sort=date:D:L:d1&ie=UTF-8&btnG=Google+Search&client=NSF&oe=UTF-8&proxystylesheet=NSF2&site=NSF&q=open+source

NSF regularly awards grants under the condition that software developed under those grants is given an open source license. Some very progressive thinking, and shrewd IP stewardship from the NSF folks, so we’ll forgive them for not mentioning open source directly in their plan.

Department of the Treasury

Yet another revelation. Treasury plans to cultivate open source projects to facilitate collaboration between agencies and between Treasury and the public. I’ve written about exactly this kind of collaboration before, back in December, so I’m enormously pleased to see that Treasury agrees. I’ve emphasized my favorite passages here:

In the areas of transparency, participation, collaboration, and flagship initiative, Treasury strives to share its efforts across Government to avoid duplication across agencies and to improve value/outcome of efforts. Treasury seeks to manifest cross-agency transferability in at least two of the following ways:

• Make training available to other agencies by opening up classes/webcasts to other agencies; providing slides, video and/or audio after the training; and posting on an e-learning platform.
• Name an advocate who gets the word out about what the agency has to share and invites other agencies to contact that person to learn from him or her.
• Design procurements for enterprise (where government is the enterprise) or in such a way that what is created can be shared across government at no cost.
• Develop and post code so it can be shared with other agencies (open source or the contract written such that the government owns the code.)
• Share platforms utilized by the agency with other agencies at no cost.
• Create participatory events across agencies with related missions.
• Collaborate on projects and challenges with the public and with the private sector in partnership with other federal agencies that have similar missions.
• Share all materials, results, tools, and training that could be transferable to other agencies with the Interagency Working Group as an efficient central dissemination mechanism.

Veteran’s Administration

The VA is an enormous consumer of information technology, and gained early recognition from the open source community for its public domain VISTA electronic health record platform. On page 22 of the VA plan, it becomes clear that the VA is expanding its use of open source to lower the barrier to entry for developers who want to help the agency:

AViVA

A Virtual Installation of VistA Architecture (AViVA) is a recent innovation that we are using to support collaboration. AViVA creates a universal user interface for the electronic health record and includes prototyping of data connectors in order to securely link the AViVA platform to patient data from any source. The AViVA project incorporates HealtheVet as an update of the VistA legacy database system.

VA’s current electronic hospital management system uses a graphical user interface known as the Clinical Patient Record System (CPRS). CPRS data is stored in the legacy data system known as VistA. CPRS requires installation on each machine that operates the program rendering the program difficult to scale and expensive to maintain and update. AViVA’s implementation improves this model in two ways. First, AViVA creates a modular, web-enabled electronic health record system that can be easily and remotely maintained. Second, Medical Data Web Services (MDWS), which can be accessed by the Department of Defense, will allow the creation of applications for any data source to be plugged into the system.

AViVA is a very exciting program for the collaboration portion of our Open Government Plan and because we are committed to creating systems that allow health care providers to collaborate to provide the best care for Veterans. AViVA’s web based presentation layer will allow our doctors and nurses around the country to search patient records as simply and succinctly as you can search for pizza on Google Maps and as securely as the best retail financial service businesses. Additionally, AViVA creates collaboration between VA and DoD, our partner in caring for our nation’s heroes. Finally AViVA creates an open source platform that allows software to be shared with entities outside of VA, creating opportunities for further innovation and development beyond the agency.

National Aeronautic and Space Administration

“NASA is working to make open source software development more collaborative at NASA to benefit both the Agency and the public,” it says right on the first page of the NASA open government plan. Here’s an agency which has always relied on a vibrant research community, software developers, and a culture of innovation. I’m not surprised by their focus on open source, but I am delighted. Among other things, NASA will be sponsoring an open source code competition, has an entire section of their plan devoted to open source development, and will be developing their Nebula cloud computing system on open source software.

It’s fair to say that NASA’s plan is the strongest I’ve seen for the open source community.

Your Turn

Who did I miss? What other opportunities for open source have you found in the open government plans? Leave a comment and let us all know!

The good people at O’Reilly have posted my Open Source in Government talk at OSCON 2009 on blip.tv. It’s also on YouTube. I’ll admit to cringing a bit when I started watching, but I’m pretty happy with how it all went. Here are the slides.

In the panel afterward, someone asked my why open source developers should be helping companies make money on open source software, or helping the military-industrial complex or the prison system. I completely sympathize. There’s no reason whatever that someone should help the military or the prison system if they don’t want to. Those were just the examples that I used. There are many opportunities to work with the government elsewhere, especially at the local level. A good way to start is by finding something that’s annoying or broken in your local schools or library, and use open source software to fix it. Open Source for America should be making it easier for people to find these opportunities. But more on that later.

As to why developers should help commercial open source companies make money, that’s some well-trod territory. But I’ll take a stab. The point of the organization isn’t to make these companies richer, the point is to make it easier for the government to adopt open source. That’s an inherently good thing. I fully expect people to make money as a result of this — that’s how we know it’s working. Open source creates opportunities and makes it easier for people to enter a market. That’s great, right?

In all, I’m happy with the talk, I’m happy with the questions I got, and I was very happy with all the great feedback I got. Thanks to O’Reilly for giving me the time.

Open standards also keep costs low for buyers, who have many options and a minimum of friction when they want to switch from one implementation to another. Because the standard is open, there is no danger of being locked into a single vendor since anyone can create a new implementation against the standard. Since open standards will always exist, there’s no danger of the standard disappearing, becoming unsupported, or being later made proprietary. An open standard will encourage these efficient, durable markets for as long as the standard is useful.

Too often, plain-vanilla standards are conflated with open standards. It’s a shame, because it’s a very important distinction. Anyone can create a standard and collect a large ecosystem around it. Microsoft Word’s document format is a great example of this. It is a standard only inasmuch as it is ubiquitous. In fact, it was only documented formally in February 2008, before which anyone who wanted to use the format had to guess at its inner workings or pay a fee to Microsoft. So while the Word document format was a standard, albeit ad hoc, it created no market, discouraged competition, and represented a serious barrier to entry for anyone who wished to create a word processor that would compete with Microsoft Word.

By contrast, the Open Document Format was developed collaboratively and is freely available to anyone who wishes to implement it. The demand for a viable alternative to the Microsoft Office document formats had found an outlet, and there are now dozens of applications which support the format, including Open Office, Google Docs, and even Microsoft Word itself.

So just because something is a standard, that doesn’t mean it’s open.

What’s Different About Open Standards

In order for a standard to be open, it has some very specific characteristics. Not surprisingly, there are many opinions about what those characteristics are, and Wikipedia has an excellent collection of these. Though there are dozens of definitions available, there is broad agreement that users of an open standard must have a few specific characteristics, which I’ve described as freedoms:

• The freedom to read and implement the standard at no cost.
• The freedom from patent encumberances, either through royalty-free licenses or a promise of non-assertion by the patent holder.
• The freedom to examine and participate in the development of the standard, through a consensus or majority decision-making process.

There are many other characteristics described by these various governments and international bodies, many of which are less definitional than they descriptive of a healthy and useful open standard. We will return to those shortly.

You could say that the freedom to obtain and implement a standard at no cost is the defining characteristic of an open standard. There is no fee, no contract, and everyone is using the same set of instructions. This freedom, though, only distinguishes open standards from closed, for-fee standards. Standards which are proprietary but available at no cost, like the Microsoft Office document formats, cannot be called open. This is at the heart of the confusion over open standards – many do not understand that truly open standards provide more freedoms than no-cost standards.

Freedom from patent encumbrances is crucial to making the first freedom practical. A prospective implementer needs to be assured that they won’t subject themselves to rent-seeking to gain compliance.

The ability to participate in the ongoing development and refinement of a standard is the most unique characteristic of open standards. It is how a standard stays open and viable over time. Open standards must have a facility for revision and correction that accommodates its community of implementers and users. If there is no facility for this, the standard is simply a mandate from one vendor or industry to others. This introduces the possibility of features that favor one implementation over another. Worse, a sudden and unexpected change in the standard could thrust existing implementations into non-compliance – essentially casting all existing implementations out of the ecosystem, which requires time, money, and effort to re-enter. This uncertainty decreases the likelihood that the standard will be useful and thus widely adopted, both now and over time. An open process removes this uncertainty and significantly improves the usefulness of the standard.

So describing a standard as open means much more than “no charge”. If a standard is meant to be a neutral set of rules by which implementations compete for users, simply being free (though useful) is insufficient. An open standard cannot be controlled by a single party, even if it is available at no cost. Without a transparent governance, the standard is simply free.

The Role of Open Source Software

I’ve framed these characteristics of open standards as freedoms to invite comparison with the four freedoms of open source software:

• The freedom to run the program, for any purpose (freedom 0).
• The freedom to study how the program works, and change it to make it do what you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to improve the program, and release your improvements (and modified versions in general) to the public, so that the whole community benefits (freedom 3). Access to the source code is a precondition for this.

Open source and open standards are the same process, applied to two different problems: open source is for computer code, and open standards are for data. The details vary, but when we talk about the principles of open standards or the benefits of open source, we’re describing the same collaborative innovation process.

Returning to the list of open standards definitions at Wikipedia, you’ll see that a few of these definitions encourage or mandate multiple implementations of the standard. This is wildly important, but I don’t think it’s definitional for open standards. Multiple implementations ensure that the standard is practical, which is not something to take for granted when complex technical standards are developed by committee. Having multiple implementations also means choice for the user, which is central to the value of a standard. So multiple implementations are great.

Now imagine if at least one implementation of this open standard is done with open source software. We not only have a freely available standard that anyone may use, but a freely available implementation of that standard that can be examined and re-used by other implementers. An open source implementation is also an “implementation of last resort” should every implementation disappear or go out of business. Those who rely on the standard have not lost their ability to interoperate. An open source implementation inoculates an open standard against the future, and ensures that there is always at least one competitor in the market. This is good for the standard, and good for the users.

Contrariwise, imagine an open standard with only proprietary implementations. This is scenario works, but substantially undermines the stated goals of an open standard: choice, freedom, and durability.

We may have many proprietary options before us, but without an open source implementation, it is significantly more difficult to create an alternative. You’re starting from scratch each time, so the bar is unnecessarily high for newcomers to the market. Likewise, our freedom to change implementations with a minimum of friction is diminished.

Ideally, every implementation would provide some assurance of interoperability that facilitates my ability to freely choose among implementations. This is not always the case, though, and it’s possible for vendors to “embrace and extend” a standard, which is code for creating an implementation that adds vendor-specific hooks which may interfere with my ability to switch to another implementation. If all the implementations are proprietary, they could all be compromised in this way, leaving me with no true implementation of my otherwise open standard.

If I have no open source implementation, I am reliant on the continued success of the proprietary implementations. If a company goes out of business or leaves the market, I may be left with no viable options. Of course, other competitors could appear – I could even write an implementation myself, though that’s often so expensive as to be impractical. This is all unnecessary if there is an open source implementation available. Again, it is the implementation of last resort.

So while I think it’s excessive zeal to say that an open standard must have an open source implementation, it’s wise to examine claims about “openness”. If there is an open standard that lacks a transparent governance, and has no open source implementation, you have to seriously consider the long-term viability of that standard, and its ability to provide a properly functioning market.