By Andy Oram
The following talk was given before a group of economics and political science professors at Senshu University in Tokyo, Japan on November 16, 2001.
Mr. Iwata has done me a greater honor than I deserve by inviting me to meet today with a group of social scientists and political scientists in order to discuss the social implications of open source or free software. I am well aware that the members of the audience are much more thoroughly educated than I am in the philosophy, economics, psychology, and political science needed to undertake this task. I deal with free, open source software all the time, writing about it and using it in my daily work (such as the research and writing of this speech). What I can do for this audience is to summarize the small amount of research I’ve discovered that tries to explain why open source succeeds. I can also contribute a few ideas that I believe to be original—at least I can assure you I have discovered them independently. The next step is up to all of you. Work by this audience will develop useful insights and train policy makers and managers to apply these insights.
The topics in my speech are as follows. First, I will summarize what free software or open source software is, and describe the typical development process that so astonishes observers. Then, because we must understand the principles that make this software successful before we can hope to apply the principles more generally, I will summarize the reasons given by the proponents of open source software for its success. Then I will broaden the discussion to other types of information, starting with a rudimentary discussion of the roles information plays in our economy and society. I’ll point out some current trends that make information in our society less open. I’ll also make a couple points about the limitations of the open exchange of information—some things it cannot accomplish. And I’ll end with a suggestion concerning a model for a better society offered by the open source movement.
First, let me give the obligatory introduction where I define and offer a history of free or open source software. I’ll try to do it in a way that’s not too boring for those who know these things.
When computer programming was invented, people at first would either keep their software to themselves or show it to anybody who was interested. It took a while before computer companies realized they could make money by selling software, not to mention the emergence of companies that did nothing except sell and support software. (Actually, the companies claim to be licensing software rather than selling it, a legal subtlety that most people ignore and that courts have even overturned, but that shows that the companies take absolute control very seriously.)
Thus started the era of proprietary software: software that you can license and use but that you cannot deeply understand or change. We may now be seeing an equally significant shift at this point in history, because fewer and fewer companies are finding it possible to survive in this business. Software companies are instead experimenting with keeping their software running on their own systems and renting the use of the software over the Internet; this trend is called web services. We’ll know the trend has reached fruition when Microsoft releases the newest version of its Windows operating system as free software. As for the software used to run a web service, it could be either proprietary or open.
Free software has continued to exist all along, but a group of programmers developed a conscious sense of community and mission in the 1980s, founding the Free Software Foundation in 1985.
Free software can be sold, but it cannot be hoarded. People who promise to make their software free may sell it, and customers may be willing to buy it because getting it on a disk or tape may prove convenient. But developers will also offer the source code to the software to anybody who wants it. The source code is what they write in order to create their programs, so giving you the software doesn’t just make you an educated consumer (although that’s one benefit of free software). This act goes much, much farther. It puts you on the same footing as the person who originally created the software. Anything they can do with the software, you can do too. The distinction between vendor and consumer is erased, and now you are equals in a community of people who care about the software.
What free software is widely used today? You probably know that it includes the most popular Web server (Apache), nearly all the software employed to make sense of domain names (which we all use to find resources on the Internet), and a pretty substantial chunk of all the software used to route electronic mail.
In addition to these impressive functions, two major operating systems are free software, along with lots of programs that run on them. One operating system is the Berkeley Standard Distribution (BSD), which introduced thousands of students to its version of Unix and became the foundation of some of the earliest commercial versions of Unix. BSD also provided important lessons and models for the other free operating system, Linux. It would be reasonable to say that BSD launched the great era of commercial Unix servers. Some might say that Linux is bringing this era to a close.
Linux has drawn more attention than any other piece of free software, partly because its rise to prominence surprised almost everybody, and partly because the story of its birth is fun to tell (but I’m not going to do that today), but most of all, I think, because it was one of the first projects to take the notion of shared development to an extreme. There are lots of people contributing critical code to Linux who just appeared from nowhere and weren’t known to anybody. Many hands make light work, and what surprises people is that they make high-quality work too.
The symmetry in relationships fostered by free software, along with the community’s unpretentious atmosphere and its trust in an unstructured process, are disconcerting and challenging to those who aren’t used to working that way, such as many traditional businesses. The different meanings of the word “free” in English foster confusion, and lots of people found that the moral connotations of the word placed a drag on their discussions with businesses interested in adopting the software. The term “open source” was therefore proposed in 1997 and publicized with some fanfare in 1998 at a summit organized by my company, O’Reilly & Associates. The Free Software Foundation and others continue to use the term “free software,” declaring that the moral connotations are precisely what we need to emphasize. This audience may well adopt the same attitude today, but I am going to use the term “open source” for the rest of this speech because it is the term used by companies in the field, including my own company O’Reilly & Associates, and is also the term used by Eric Raymond, the author whose writings are cited most often in the field.
In this speech I will cite these essays by Raymond, which are for the most part available online but are also published by O’Reilly & Associates as a conventional book called The Cathedral and the Bazaar. In addition, I’ve found some interesting social research about open source in an online journal called First Monday. Raymond, incidentally, warns that it’s too early to apply insights about free software development to grand social issues. If he is tacking so close to the shoreline, I am certainly not going to venture out onto the ocean. My comments about society will try to provide a context for other people’s research, rather than apply ideas directly from the open source movement.
Let us move from a description of the open source phenomenon to the first big question: why is it successful? One must answer this question on two levels. First, one must ask what motivates people to contribute their efforts to a project in the absence of material rewards. Second, one must ask why the results are good. Why do users want to use it, and why is the quality so high that it often triumphs over proprietary software in the market?
Therefore, let’s move to motivation. There are two types of explanations offered as to why people contribute to open source software: practical explanations and psychological explanations. I’ll start with the practical ones because they’re somewhat more interesting—although I have a psychological explanation of my own that may spark some discussion.
First, if programmers release their software with no restrictions, they remove barriers to distribution and adoption, thus creating a much larger community much more quickly than they would have if they encumbered users with licenses and fees. Open source software may be the key to a major goal sought in most businesses: building market share. It’s interesting that in the 1980s, when American businesses became frantic over Japanese competition, many American analysts claimed that the large Japaneses manufacturers were successful because they concentrated on building market share instead of maximizing profits. I have no way of verifying that claim, but open source developers are taking that strategy to an extreme.
A leap of faith is required of open source software users, because programmers naturally have to minimize their own burdens of liability and support. The programmers will have to offer a lot of support to be taken seriously, but they’ll discover that their users will jump in and help.
That leads to the second practical motivation for open source releases: users return excellent error reports and occasionally even fixes. As Raymond reports, giving users the source code vastly improves the quality of their error reports. Let me explain why by providing an analogy.
Suppose that, at some point in your academic year, students start coming late to some of your classes. You can never predict who will come late; sometimes the classes are full when you begin and other times 20 or 40 or even 60 percent of your students are late. Very annoying.
Suppose you talk to these students and find that they all take the subway to class and have to pass through the Otemachi station. Suppose also that, by reading the newspaper or phoning the subway service, you discover that the Otemachi station is undergoing repairs and that trains are slowed down at certain times of the day.
Now you are no longer the helpless victim of schedules. You know exactly who will be late, and when. You can also ask students to alter their travel plans so they will no longer arrive late while repairs are underway. By becoming privy to the subterranean activities of the Tokyo subway, you are empowered. By providing source code, open source projects open up their own subterranean activities, which is equally empowering to the users of software—at least those with the basic programming knowledge to read it.
Another practical aspect of releasing software as open source needs to be mentioned: the programmer might as well make it open source, because deriving money from a proprietary software business is extremely difficult. Very few programmers have the talent and energy to start a software company. O’Reilly & Associates was actually in the software business for several years, and we were making a profit too. Yet we didn’t make enough profit to find it worthwhile to stay in the business. Open source software can be the foundation of a business, of course—Turbolinux is an example of a company that has a strong presence in Japan, selling and supporting open source software. But a company is not a requirement for developing and distributing open source software. One can release it by putting it on a Web site and posting to a mailing list. Open source presents an irresistibly low barrier to entry.
In short, open source software is a highly rational proposition. Nevertheless, people also seek psychological answers to the question of why contributors participate.
Open source contributors enjoy programming, of course, but that doesn’t explain why they release the results of their programming to the world. Most people who try to find a justification for the generosity of open source programmers say that they want to enhance their reputations. Raymond says that the term “ego-boosting” is used by many such programmers to explain their love for sharing their code. Lots of authors on the topic talk about potlatches and gift economies, and I will return to this theme at the end of the talk. But I believe I have found another psychological source of satisfaction, one that looks quite plausible.
My theory draws on the work of Joseph Weizenbaum, a computer science of some historic importance at M.I.T., and the author in 1976 of the book Computer Power and Human Reason. This book, astonishingly enough, seems to be out of print, even though it has been extremely influential for anyone who tries to think deeply about the impacts of computers on society.
Weizenbaum was not particularly kind toward his fellow practitioners. He spoke in his book of the “compulsive programmer,” which is an exaggerated portrait of what people in the computer field usually call a “hacker.” Let me read some passages of his book concerning the motives that drive this particular type of computer programmer:
(pp. 112-113) The computer…is a playing field on which one may play out any game one can imagine. One may create worlds in which there is no gravity, or in which two bodies attract each other, not by Newton’s inverse-square law, but by an inverse cube…law, or in which time dances forward and backward in obedience to a choreography as simple or complex as one wills. One can create societies in whose economies prices rise as goods become plentiful and fall as they become scarce, and in which homosexual unions alone produce offspring. In short, one can singlehandedly write and produce plays in a theater that admits of no limitations….
(pp. 115) An engineer is inextricably impacted in the material world. His creativity is confined by its laws; he may, finally, do only what may lawfully be done…The computer programmer, however, is a creator of universes for which he alone is the lawgiver. So, of course, is the designer of any game. But universes of virtually unlimited complexity can be created in the form of computer programs. Moreover, and this is a crucial point, systems so formulated and elaborated act out their programmed scripts. They compliantly obey their laws and vividly exhibit their obedient behavior. No playwright, no stage director, no emperor, however powerful, has ever exercised such absolute authority to arrange a stage or a field of battle and to command such unswervingly dutiful actors or troops.
One would have to be astonished if Lord Acton’s observation that power corrupts were not to apply in an environment in which omnipotence is so easily achievable. It does apply.
In other words, Weizenbaum is saying some computer programmers—not all—are attracted to programming because it gives them absolute power over a part of the universe, and they love this power even though that part of the universe exists only in a small part of a silicon chip during the time interval between the initialization and termination of their program.
Weizenbaum holds a personal disgust for this kind of programmer. From psychology courses I have learned that this opprobrium is probably unjust, because the unconscious wish to be omnipotent stems from the infant stage of life and lies dormant to a greater or lesser extent in all of us. Aside from moral judgements, Weizenbaum draws from his observations some very deep and insightful conclusions about the way society misunderstands the role of computer programs and expects more from them than they can deliver. That is not the subject of my speech, though, so I’ll turn back to open source and report some good news. I believe that many programmers with this will to power have found a healthy and productive way to sublimate the drive.
Think of it this way. If I write a nice program that performs useful calculations and run it once a week to serve my personal needs, I experience the joy of power over the computer for just that one-eighth of a second that the program takes to run each week. But imagine, now, that I release it to the public and get my friends and colleagues to run it. Imagine that it turns into something really useful that thousands of people run. Or gets loaded onto servers in organizations around the world to handle millions of hits across the Internet each day. Thanks to all these other people who use my program, I can glory in having far more power than I ever could attain during my personal runs of the program. By relinquishing control, I gain psychological power. I can log into SourceForge (one of the most popular repositories for free software) and watch the downloads of my software.
So the free software movement rewards the unselfishness of the programmer while perhaps satisfying a deep psychological need for power and control that he or she, like all of us, possesses.
At this point I have to orient us all in the speech. I have tried to answer one-half of the question with which I started my inquiry: the question as to why people contribute to open source software. The other half of the question is why the software turns out to actually be good.
Here, explanations are fairly uniform across many commentators, and most of the explanations are well stated by Raymond.
First, people are motivated to reach creative heights when they work out of personal interest, not for material reward.
Second, development can proceed in parallel. This is clearly valuable if the project can be broken into many subprojects. But it’s useful even when it’s redundant effort—that is, when several programmers independently try to solve the same problem. The community gets to choose the winner, so superb code becomes the norm if a project is popular enough. Such redundancy may appear to be a waste, but it is the same source of competition that represents the great strength of capitalism. Furthermore, redundancy comes to a halt in open-source development long before it does in proprietary software development.
Parallelism can be viewed in another way as well. One researcher (Kuwabara), writing in First Monday, says that distributed development is also a kind of parallel search that is effective because it finds its goal quickly.
Widespread examination of source code expedites the discovery and repair of software errors. This claim for open source has been disputed, because distributing code to 1000 users could simply means that the same bug is found 1000 times. Meanwhile, a serious and subtle security flaw may remain hidden for a long time. But the speed with which the open source community fixes bugs that have been found is legendary.
The most inspiring aspect of open source development, and perhaps the most fruitful for social science research, is the transparency it fosters—a form of instantaneous and supremely honest communication. No one can hide a bad decision because source code can be examined by experts everywhere. Furthermore, these expert reviewers risk nothing by reporting problems and disagreeing with the project managers. Contrast this to the employees of a commercial firm, who are constrained by their desire to look good to superiors.
Socially, the atmosphere on an open source project’s email list is instructive. Where there is fully open communication there can be no control hierarchy, and where some people control others there cannot be fully open communication. How many wars and revolutions were sparked by rumors? I believe that the 1789 French revolution and the 1871 Franco-German war are examples. In open source projects, most disputes are resolved by consensus. The famous “fear, uncertainty, and doubt” (FUD) in the commercial world are less evident. But these projects are no Gardens of Eden; I’ll return later to the evidence of occasional dissension.
One article by George N. Dafermos in First Monday suggests that widespread feedback over the Internet helps open source developers stay focused on their users’ goals. Many commentators say that a focus on the whole “value chain” is key to success in modern business. If I am a manufacturer of nuts and bolts, I should not be satisfied simply to sell you 10,000 nuts and bolts. I should find out what you’re building with those nuts and bolts and what the purpose is of the final machine. I should be helping you improve your product—even if that means you need only 5,000 nuts and bolts this time—and I should be using your feedback to improve my product, because in the long run we’ll both make more profit. Perhaps the open source community trains developers in this kind of forward thinking.
It’s hard to imagine why users would choose proprietary software for anything where a mature open source alternative exists. The price is the least of the advantages for commercial users. More important is the convenience of installing open source software as needed, without having to count users and deal with a proprietary system’s license manager, which itself can be a source of errors and denial of service. With open source software, if you find a bug, you can fix it. If you want a new feature, you can add it. These activities are “frictionless,” to use a term favored by business analysts.
The difference between proprietary software and open source software in the area of configuration has deep ramifications. Proprietary software can evolve and provide lots of options, but it will never fit the user’s needs like a glove, as open-source software can. The ability of users to create a product all their own is sometimes cited as a reason Linux is popular in embedded systems, where it’s critical to minimize the size of the software and provide only what’s really needed.
When analysts discuss ease of communication and product distribution as factors that favor open source software, they always stress the role of the Internet in allowing people around the world to collaborate closely. The implication is that the Internet reproduces the advantages of putting innovative people together in a small space and making them work in close proximity for months at a time. I believe that the Internet provides even more advantages. Even if you took all the people developing Linux from around the world, made them live in a single building, and let them work side by side, communication might well be less effective than using the Internet as they do now. The reason is that the Internet keeps a record of everything. Someone trying to figure out a chain of reasoning, or joining a project late in the cycle, or simply interested in history, can reconstruct what happened.
I have spent a great deal of time laying out the social and cultural aspects of open source development. What you want to do, I know, is talk about its social and cultural implications for a larger society. The popular urge to draw parallels is strengthened by the perception that we live in an “Information Economy,” where information has replaced manufacturing as the main lever of productivity. So if we want to explore the importance of open information, we should take a moment to establish the economic role of information.
Many of you, like me, may be tired of hearing about the Information Economy. After all, we cannot survive by eating information, or wrap ourselves in information to stay warm, or burn information to power our appliances. My perception is that we live in an economy of food, clothing, and fuel, the same basic economy we’ve had for fifty thousand years. All the value in the economy comes from extracting, building, and transporting material things; information merely makes us more efficient at that.
My point of view, of course, is derived from the labor theory of value laid out by Karl Marx. He pointed out the centrality of extracting, building, and transporting materials. Nothing else can add value. But a lot of things can diminish value. These include the costs of setting up purchases, of bookkeeping, and of storage. You can reduce such costs by having better information—and there’s the information economy right there, in this tiny chink of optimization of unproductive activities.
Most people don’t agree with me, of course. Many prominent people in society believe that information has value of its own and is a new source of social wealth. And these people invested billions of dollars in snazzy web sites for companies whose role was to be information intermediaries.
Marx, who was a life-long learner of prodigious proportions himself, could be interpreted to allow a somewhat greater role for information in the economy. Information could be part of the labor embedded in machinery, which reduces the costs of production.
An excellent explanation of information’s role in production can be found in a recent article from The Atlantic Monthly called “The New Old Economy: Oil, Computers, and the Reinvention of the Earth” (January 2001, pp. 35-49). In this article, author Jonathan Rauch shows that traditional mining and manufacturing companies make extensive use of information, but not by providing a Web site or online customer help desk. They make it a part of the key processes that provide the reason for their existence. Better oil exploration techniques, for instance, have allowed the average oil well to yield four times as much oil in 1998 than it did in 1980. Explorations are much more likely to be successful these days. Known oil reserves in the world are one and a half times the known oil reserves of the 1970s, and ten times the known oil reserves of the 1950s. This news may be bad for global warming, but shows the incredible increase in productivity that information technology can achieve.
This role is where information is crucial to our economy. Open source software is not much help in oil exploration, though. Is some programmer going to be seized one day with the urge to take time off from work and write an imaging program that displays the seismic data found by oil company surveys? Actually, it’s possible. Seismic data is useful for many social purposes, as the Japanese know very well, and I could imagine some geologist releasing to the public a program that handles that data. Such a program might also be useful in archeological searches, as reported by an article in yesterday’s paper. But it would be only a small part of the software needed at an oil company to do the activities reported by author Rauch.
So each oil company is going to develop its own oil exploration software, and it’s not going to release it under an open source license. To the contrary, because this software provides a competitive advantage, it will remain one of the company’s most closely guarded trade secrets.
One can imagine a socialist economy where there is no proprietary reason to keep oil exploration software secret, but this does not change the basic economic imperative for developing the software. Some socialist manager, whether a philosopher-king or an elected representative of the workers, must decide that developing the software is worthwhile and ask a programmer to do it. All in all, only a tiny percentage of software offers benefits to any organization or person besides the one that originally developed it. This is confirmed by Raymond.
My conclusion from this exploration is that the social phenomenon of open source is limited to certain areas of software development and has little application to the economy as a whole.
But we’ve spent enough time on the economy. Information has an equal or greater importance outside of economic activities. It’s the foundation of academia, to which the open source movement is often compared. Bill Joy, a key developer of BSD and now Sun Microsystem’s chief scientist, says that the Berkeley computer researchers released their code to the public because research results have always been released to the public. Who who imagine doing research and keeping it secret?
The admirable role that universities play in publicly sharing their knowledge arose because universities became established long before people raised economic development to the highest goal of society or considered the economic benefits of thought and discourse. But now universities are fencing off information; the idea of doing research and keeping it secret is no longer at all absurd.
First, universities started accepting more and more corporate funding for specific projects. Then they allowed their professors to spin off academic research projects and to profit from the resulting commercial ventures. Then the U.S. government relaxed its traditional restrictions on how much universities could patent and commercialize publicly funded work. The trend is clear, and it’s downward. We are turning into a society where we have to beg for the most advanced research information to be released to the public, where we can no longer take this for granted.
Outside of universities, there is also a large public sector for information. In most countries, a wide range of information is publicized by the government where it can examined by independent, public-interest groups. For instance, environmental watchdog groups benefit from knowing what hazardous chemicals are used in factories and other facilities around the country. However, this information has recently been removed from government publications, because it could be used by terrorists to find vulnerabilities.
In many countries, for the same reason, agencies throughout government are racing to hide information that used to be available to the public. In the short term, this is a responsible precaution. In the long term, it could be disastrous. Certainly, I don’t want terrorists to exploit information about hazardous substances to launch an attack. But over time, once the information is hidden, the guardians of such hazardous substances will become lax and raise the risks of leaks, workplace exposure, and accidents. Transparency has its dangers, but important benefits as well.
Open source offers few generalizations about information, but I think it reinforces some things we already know. For instance, it shows the limits of rationality and objectivity. Even in the case of software that can be tested under controlled conditions, over and over, what is good is not always obvious or subject to objective verification. An open source community can resolve a lot by consensus, but sometimes a leader must step in and make a decision.
Furthermore, rationality is often compromised when people have invested a lot of effort in one system. Watch when they argue about their favorite computer language or operating system. The community has aptly applied the term “religious war” to arguments over the relative benefits of Java and C++, or of KDE and GNOME, or many other such competitive software systems.
I will end with the most brazen speculation of all, but I will keep it short and inconclusive. Can open source offer a peek at a better society? A possible answer is suggested by the common notion of open source software as a gift culture. Raymond writes:
Gift cultures are adaptations not to scarcity but to abundance. They arise in populations that do not have significant material-scarcity problems…We can observe gift cultures in action among aboriginal cultures living in ecozones with mild climates and abundant food. We can also observe them in certain strata of our own society, especially in show business and among the very wealthy.
And later he describes open source as the expression of computer developers who have material plenty and don’t want to participate in any more “command-control structures” or “market-style exchanges.” To summarize the observations presented earlier, this open source development culture is based on:
a free and instantaneous exchange of all available information concerning product features, problems, and available resources;
evaluation as much as possible by the entire community, with intervention by leaders where ambiguous information makes it necessary;
a lack of hierarchy or control relationships among individuals; and
trust that the proper result will emerge from an unstructured and undirected process.