Contents of this chapter
Before entering the computer field, I spent a year trying to earn a social work degree. While counseling authors at O’Reilly, I often felt I had opened a wormhole and transported myself back to that social work program. I dealt with the pressures of authors’ work, pressures of their loss of work, conflicts with urgent family needs or other time commitments, struggles with their depression, and everything else that comes into the life of a human being.
Such patience proved valuable during my later years, when O’Reilly managers often plunked me down into projects that I had no part in planning. Many of these projects had lacked oversight earlier, as they crept their way to a dead end. I was often asked to fix a mess, sometimes at breakneck speed, and without much guidance.
One sad book project brought in a couple networking experts who had created quite successful presentations on that subject and had been recruited by O’Reilly to do a book. It quickly became obvious that they fell abjectly short of fashioning a coherent prose narrative. But no one at O’Reilly had a longitudinal view of their work—a problem I could never imagine happening during my first 10 or 15 years at the company. Instead, junior editors were assigned at various meandering stages of the authors’ writing to suggest changes. Each editor recognized and tried to react to the multifaceted problems, but with a different perspective, using different words. The authors couldn’t make progress because they kept getting hit with dire assessments, always shifting.
Somehow, these poor fellows produced a complete draft. It even went through a tech review that highlighted some of the same organizational problems that the editors had clumsily tried to address. So numerous people—not just the authors and O’Reilly staff—had been inconvenienced by this misdirected project.
To render the situation even thornier, a sponsoring company had read early versions of the muddle and found useful material in it. It continually amazes me how people can approve patently unviable writing. So O’Reilly had already made money licensing the early versions of some chapters, making it even harder to end the project.
Trapped in a situation with no redemption, I treated the authors with respect and engaged with an intensity that no one had granted them previously. After discovering that they could not place their ideas into a sequence that readers could follow, I started writing whole chapters by myself. This is something I had never done for an author before, and I knew it was ridiculous but felt I had to do whatever I could for them and for O’Reilly.
Luckily, I was rescued by my boss at that time, Rachel Roumeliotis. Roumeliotis had not been at O’Reilly for long—at least, compared to the people with whom I’d worked at the beginning of my career there, whose tenures spanned geological epochs—but she rose remarkably fast in the organization. It seemed like she was always taking on more and more. For instance, she planned several conferences and started new ones in addition to all the book work. (This made sense in our integrated media strategy, where all our activities intertwined.) She was much younger than me, but happily accepted a stack of old 12-inch LPs of classic 1960s rock music from me when I was trying to clear out my basement.
So Roumeliotis laid down the law: it was not my job to write a book for the authors, and if they could not complete the job in a satisfactory manner, the project would come to an end. Although the authors expressed understandable frustration at the termination of a project they had been permitted and even encouraged to drag out for so long, they remained open to doing other kinds of material for the company. I was thankfully relieved of further responsibility.
Another book about the same time provided a happier outcome. This book was also on the cusp of cancellation after having been denied expert attention. The half dozen chapters produced so far on data management lacked discipline and direction, but I sensed some nuggets of value that steeled me to save the book.
Unlike the authors on the project previously described, this author could listen to editorial advice and pull together his work. We worked together without blame and without resistance to identify the concepts that required emphasis and wormed out the valuable material from his edifice of verbiage. As is typical of disorganized books, this one was full of redundancy. Reducing the material to essentials was highly gratifying. The book was ultimately finished and released to some acclaim, along with enthusiastic support from a sponsor.Throughout my career at O’Reilly, I handled both projects that sprang from my own convictions and projects handed to me by others. Both categories included great successes and embarrassing failures. The best stories from these writing projects form the subject of this chapter.
Regional reports (2016)
Not all my projects at O’Reilly concerned mind-numbing technical issues such as data storage—although I always tried to bring alive even the most mundane topic. I landed a few lavish opportunities for creativity, including assessments of computer tech opportunities in different cities or countries in 2016.
Roumeliotis told me the company was considering a series of reports on promising tech regions, a bid to get attention from people living in those regions and increase our followers and conference attendees. I went through some twenty years of files on authors and tech reviewers to find people who made their careers in places as diverse as Ottawa and Lawrence, Kansas. I turned up lots of detail through my interviews and stored them all in files that I turned over to Roumeliotis. None of it got used, but she did give me a few assignments.
Roumeliotis thought regularly about appealing to new demographics. This was around the time she moved our Open Source Convention from Portland, Oregon to Austin, Texas. She thought maybe we had maxed out our exposure in the Pacific Northwest and would attract a lot of new people from the Midwest by relocating to the middle of the continent. Our first year in Austin (which I attended) drew a pretty good crowd, but in the second year (which I didn’t attend—the first Open Source Convention I ever missed) the draw wasn’t gratifying. Roumeliotis took the convention back to Portland.
My part of the strategy, which involved writing reports to drum up business in new areas, also failed. But it was very rewarding for me, and I went much deeper into the subject matter than I think anyone expected. It provided me a cultural odyssey.
First I was asked to write about London, where the financial (“fintech”) industry anchored a strong computer sector. I drummed up interviews from a number of contacts, and learned about interesting differences between the English and American views of business. Essentially, U.S. venture capital is distorted by a “unicorn” model, hoping that every company will become a billion-dollar firm in a few years. The English are more patient, looking for gradual, organic growth. This and many other insights went into my 16-page report.
By extraordinary (in all three English syllables) luck, I had planned a personal trip to London with my wife during the time I was writing the report. All sorts of local references and joking puns about London life made it into the report, along with insights about the difficulties of living there. I held a meeting in a pub with some of the people I had interviewed for the report, which led to some networking that was gratifying to see. I also took some pictures of 3D-printed objects in the Victoria & Albert Museum to flesh out a short section of the report on technology and art; I had tried unsuccessfully to get interviews with Londoners in that area.
I put even more of myself into a report on Brazil. Roumeliotis had asked me to cover open source throughout Latin America, but after several interviews I determined that there was little to say about it outside of Brazil. As the largest Latin American country, and one that has cultivated its tech sector diligently over several decades, Brazil has become a center for free software development. It’s no coincidence that I had lots of contacts there. I also got a feel for its free software scene after my attendance at a FISL conference and my work with Marcelo Marques on the Hackerteen comic book I’ll describe later. I had picked up a bit of Portuguese to prepare for my visits. So I persuaded Roumeliotis that the report would be much more useful and focused if I stuck to Brazil.
I worked hard to find out what was going on with free software since my previous visits, and to be inclusive of marginalized populations, particularly women and small towns outside the major metropolitan areas. I got wonderful interviews and very dedicated reviewers, including my Hackerteen friends and Jon “maddog” Hall, a former technologist at Digital Equipment Corporation who made it a personal mission to promote education and free software in Brazil. I also worked into the report references to relevant books and movies.
In a coup de grâce, I decided to start each section of my 19-page report with a pertinent line from a famous Brazilian pop song. I had always enjoyed the pounding Samba music and the more polished bossa nova, and I knew the names of the most famous Brazilian songwriters. Internet searches readily turned up a huge range of songs, and between my minimal knowledge of Portuguese and some dictionary work, I picked an apt citation for each section.
It was these cultural references to pop songs that won over a Brazilian translator who was recommended by one of my reviewers. I got O’Reilly to approve a thousand dollars to pay for her translation, all in pursuit of getting our name in front of more Brazilian professionals.
All these reports called for a stretch in expertise that many writers would eschew. Caution is one of the chief enemies of strong writing. (Tact is another, as I mention elsewhere.) To compensate for my lack of caution and tact, I do a lot of research. I get reviews from authorities in the field, whose email addresses I hoard like precious stones. But most of all, I create a compelling argument that gives urgency to my claims. Lack of caution and tact doesn’t force a writer to adopt extreme positions or ignore counterbalancing arguments. As my Platform Independent columns illustrated, my writing tried to be radically and fervently level-headed.
As said before, the London and Brazilian efforts didn’t seem to pay off. I was put on one final regional project: an overview of the Austin tech scene. I found lots of fascinating historical background, and some disturbing evidence that Austin was starting to suffer from the same economic polarization and social disintegration as San Francisco, Boston, and other tech-friendly cities (London among them) caused as affluent computer people took over neighborhoods and raised rents beyond what other residents could afford. My research brought some valuable contacts that could have assisted us in further networking.
I proudly turned this brief report over to Roumeliotis, who informed me that it didn’t meet O’Reilly’s PR goals and therefore would not be published. So I put it up on the popular Medium blog site. The company’s concern with Austin ended when we moved our conferences out.
Another commendable but ultimately unsuccessful O’Reilly venture offered me a novel chance to contribute to a new field—but also a hefty dose of frustration.
Our management saw the effects that digital technologies were having on manufacturing and product design. Companies in these spaces are normally less tuned-in to the potential of data analysis than industries such as finance and defense that have to keep a fine competitive edge to survive. But with the rise of machine learning and the new streams of data afforded by the Internet of Things, manufacturers were starting to see a benefit to digitizing old processes, and O’Reilly wanted to be there to push the transformation.
As with most new initiatives at O’Reilly, the main offering was a conference. I already saw this with health IT and web performance. Conferences help to create community and raise awareness, much more than a book, and if they’re successful they rake in a lot more money than books. This is the central tragedy behind the ultimate destruction of the conference group described in the first section of this memoir. Not only did conferences bring in revenue; they lay at the core of O’Reilly’s strategy for exerting influence.
Analytics in manufacturing were championed at O’Reilly’s Solid conferences in 2014 and 2015. Demos ranged from enhanced CAD tools to augmented reality. Keynoters laid out glowing visions of the future of manufacturing. I interviewed people from different continents about their innovative approaches to product development.
The venue was a major part of the fun: an old set of industrial buildings (actually a converted military base) called Fort Mason, near the Marina on the north side of San Francisco. The city had converted these buildings into a viable conference space, and the sense of connection to a blue-collar past added a sense of reverence to attendance.
Manufacturing is considered a “vertical” in business jargon. It’s hard for publishers to make money on verticals. A conference on data in general, or a book on networks in general, will generally bring in more revenue than one focused on data in manufacturing or networks manufacturing. After two years, Solid was discontinued.
But along the way, I got a cool gig for a few months. Somebody in marketing contacted me in March 2015 to say O’Reilly had promised an article to a web site run by the International Manufacturing Technology Show. The deadline was alarming—they wanted me to submit four or five hundred words in about 48 hours. Moreover, our marketing person could not provide any of the answers to the questions that a writer needs to ask, such as who the audience was or what IMTS was looking for. I had never heard of them, but I thought up a creative idea about the four criteria for judging the value of data (integrity, timeliness, and so forth) and delivered a pretty decent piece. Although I didn’t get any feedback from O’Reilly or IMTS, they gave me the impression that they were satisfied. I did not judge the marketing person for the slipshod manner in which I had been approached. Not knowing the background, I ignored the signals of a poorly managed project.
A month later, another email from the same marketing person. “It is almost time for our next submission to IMTS…” I realized that the commitment she was expecting from me was not a one-off piece, but a monthly series. She had just done exactly what she had done the month before, hitting me broadside with a last-minute request.
So this time I tried to inject a little discipline into the commission. I insisted on a phone call where marketing would lay out what they wanted and some basic parameters about the articles I was going to produce. I never got any guidance, but I quickly threw together another acceptable article (this one about robots in manufacturing) and settled down to plan my series.
Believe it or not, I was having a really good time with the IMTS articles. I found a couple blog sites about manufacturing, which gave me some interesting background and suggested to me that my subjects of interest were not being covered online. I wrote about supply chains, predicting monitoring, and ecological concerns.
I never heard a peep from anyone at IMTS; I simply tossed my articles over a virtual wall to the marketing person at O’Reilly. A few months later, though, I noticed that she was not responding. Repeated email went unanswered.
A few more months passed. Suddenly I heard from a new marketing person. The former one I’d been dealing with had left under a cloud—I suppose her managers noticed the same behavior on other projects that I had suffered from on the IMTS articles. The new marketing person apologized for ignoring my last few email messages. She had no idea O’Reilly was working with IMTS and couldn’t fathom why I was sending articles. The transition was obviously handled with the same conscientiousness as the writing project itself.
Now I had an ally again within O’Reilly. But a new barrier presented itself: our marketing contact at IMTS had also left the company, and no one was picking up the articles I had sent. My marketing person assured me that someone new would be hired and that IMTS would resume publication. That was the last I ever heard about the project. I didn’t even get the dignity of a formal ending. The bitter taste left by this project, unfortunately, was a familiar one. We’ll soon see another version of the same script in play.
Still, I had a good time covering manufacturing for a few months. The project allowed me to explore new areas for the application of computer technology. And it allowed at the least the illusory feeling that I was doing something valuable for both companies and their readers. (My articles can no longer be retrieved from the IMTS site.)
In the cozy field of health IT, many people recall O’Reilly’s involvement with pleasure. But the initiative has been almost totally forgotten outside this group. At its peak, the health IT effort at O’Reilly encompassed an annual conference focused on this subject as well as a track at our flagship Open Source Convention, a book series, an open journal on biohacking, and a steady stream of blog postings and video interviews that I mostly provided. But the company eventually found that they could not make enough money in this area. Like manufacturing, health care turned out to be an underperforming “vertical”.
I came to health IT out of the same curiosity that led me to explore Linux or peer-to-peer. I can trace my first health IT blog posting to 2003, when I covered the company Vocera as an interesting combination of database technology, 802.11 networking, and voice recognition. I recognized health IT as a special segment of interest around 2009, which is also when it came to the attention of O’Reilly management.
My initial advice to the O’Reilly managers and marketing team involved a lot of reorientation: recognizing the abbreviation CMS as Centers for Medicare & Medicaid Services instead of Customer Management System, and refraining from saying that health care is waiting for its killer app. Because health care institutions in the Obama administration were among the first and most sophisticated proponents of open government, Tim O’Reilly was familiar with many health care reformers as well. One of his particularly notable and valuable connections was Todd Park, who moved from the position of CTO in the Department of Health and Human Services to being CTO of the federal government as a whole. In short, Tim and I were on parallel tracks in the health care space, investigating the potential for our company’s health IT work and serving as advocates for technological change.
Because O’Reilly always joined a community and entered a technological area by holding a conference, management decided to extend the successful concept of a data conference to health IT. Because they were running several conferences called Strata, they started a narrowly focused one named Strata Rx. It was a critical, but not financial, success. I was told that attendees were very excited by the content, but spent their free time making connections in the hallways and mostly ignored the vendors. Of course, vendors can’t justify coming to a conference where they spend their time staring blankly into the busy hallway. So we didn’t have the money to continue.
The same goes for the three books I edited on health IT: well-received, and even regarded as indispensable contributions to the field, but not actually bought that often.
The collapse of our health IT venture came in early 2014 just when I was putting my final touches on my flagship work in this area: a comprehensive report covering everything in the health IT space from medical records to devices to research. At 50 pages, it’s the longest single project I ever did at O’Reilly.
My only disappointment with the report itself was a complete lack of graphics. I asked numerous contacts for photographs of them or their colleagues at work, but the notorious reclusiveness of the health care was interposed. Most recipients of my request cited some kind of need for confidentiality. I did receive one photo, but it was lousy. The production staff at O’Reilly tried to be helpful, giving me access to an enormous archive of stock photos they had licensed. But all the photos related to health care were glossy marketing images of clean, young, attractive people smiling over their computers or in their examination rooms—completely the opposite of the message in my document, which faced difficulties and disparities in the health care field head on. So I finished the report with no images. There is perhaps some bigger message in my coming up empty when seeking images that reflected the reality of health care and the role of IT in it—the field might not have been ready for the transformation I was proposing.
The report had been blessed by management a few months earlier, and I turned it over to production with the expectation that we’d use it to give a boost to the health IT effort. I don’t think I knew that the end was near.
During the brief time the book was in production, the company decided to forgo further Strata Rx conferences and to dismantle the Strata Rx team. Julie Steele, the editor who had run Strata Rx, took a job with another company. By the time production finished my report and released it, not only did the company totally ignore it—the company lacked any staff or processes that could have given it attention.
Thus, although the report was officially released by O’Reilly and put up for download, no publicity went out—no press release, no tweets on social media, nothing. I wrote directly to Tim O’Reilly, figuring that between his long interest in health IT and his friendship for me, at least he’d use his powerful social media presence to mention it. But he didn’t.
I was prepared to exploit my own wealth of connections and channels in health IT. I painstakingly made a list of the hundreds of people whom I’d talked to about health care over the years, and sent them copies of the report. I carried out a personal media campaign, contacting media outlets throughout the country and major medical institutions. A handful of people promised to read the report, including some who could have been influential, but I never heard back from the most important ones. The feedback I got from people who did read the report was that it was accurate and valuable, but too long at 50 pages for most people to invest time in.
And thus the health IT effort at O’Reilly came to a disgraceful end. I kept blogging for other publications, and even spun my report into a 45-minute presentation that I delivered twice, thanks to my contacts (once at the nursing school at the University of Massachusetts in Boston, and once at a health IT firm). Occasional later efforts by other O’Reilly editors have produced scattered articles and other works of interest in health IT. But O’Reilly wasn’t alone in finding disappointment in the health care space. The whole promise of the 2009 bill that kicked off a great reform effort in health care (unconnected to and long preceding the Affordable Care Act), remained unfulfilled at the time I write today. The willful refusal to coordinate care and share data has probably contributed to many deaths during the COVID-19 period, while the nearly instantaneous adoption of telemedicine to cope with physical distancing shows how the field could have adopted beneficial reforms earlier, had practitioners cared.
We’ve seen, during the course of this memoir, that O’Reilly has passed through periods of intrepid experimentation. Someone who came up with a well-considered proposal that could open up new possibilities, no matter how unconventional the proposal, could get the resources to try it. I myself have pushed the limits of O’Reilly’s publication strategy a few times.
The most way-out project I ever did was a comic book called Hackerteen. The book’s origin was as strange as its concept: I found the author on the show floor of a Brazilian free software conference.
Here’s how I ended up there: one of my authors was feeling guilty because he wasn’t meeting his deadlines. Being active in the Brazilian free software community, he made amends by getting me a speaking gig at a major free software conference called FISL.
A tight-knit group of North Americans and Europeans came for the conference to Porto Alegre, described by “maddog” Hall as a significant technology center. We were all housed together in a hotel. I knew a good number of these free software programmers and advocates, so it felt like summer camp to be with them. Furthermore, many attendees, including Hall, attended the conference year after year. Hall was one of the first North Americans to discover a strong free software movement in Brazil, extending to the highest reaches of the leftist Partido dos Trabalhadores that ran the government for many years. He came to Brazil regularly, where he helped erect training and advocacy groups.
My talk was one of a series that I had given in various places on community documentation, the term I created for volunteer-produced manuals and other help in free software communities. This subject got me a lot of traction. I delivered talks on community documentation in the O’Reilly booth on the show floor of the Open Source Convention, then at a formal session at that convention, at a Google facility in Tel Aviv and an Intel facility in Jerusalem during my first trip to Israel, and now at FISL.
The conference organizers tried to plan everything we needed, including a bus from our hotel to the conference, but time and chance happens to them all, so our bus was unreliable. My own talk was the first of the day, and the bus brought me to it late. Very few were in the audience, but otherwise the talk went well enough. On another day, with no bus on the horizon, four of us from the hotel agreed to share a cab to the conference. Richard Stallman was one of the crew, and I got a rare chance to hear his reasoning, as I discuss elsewhere in this memoir.
I visited all the booths on the FISL show floor and made some valuable connections, including a meeting with a Brazilian publisher who wanted to translate some O’Reilly books. But the most fruitful introduction was to Marcelo Marques, the heading of a training company called 4Linux. I may have been introduced to Marques by the author who brought me to FISL, or may have just struck up a conversation because he was a very sociable guy. As he described the education 4Linux offered to young Brazilians who hoped to forge successful software careers in free software, I noticed some interesting cartoon panels on the walls of his booth. The panels had striking young figures in bright colors, and I saw intriguing bits of storyline in the Portuguese speech bubbles.
Marques explained that he had engaged an artist to make the cartoon panels as a catchy marketing campaign. There was no actual comic strip or comic book, but I encouraged him to create one. I thought this would be a wonderful new departure for O’Reilly.
From time to time, editors at O’Reilly would talk of the vast, promising market represented by children and teens interested in computing. Rather like tobacco and soft drink companies—although with positive social values—we wanted to capture our audience while they were young. Nothing ever came of this perennial discussion, but the book I worked out with Marques came closest. O’Reilly managers agreed with us to try the comic book as an educational prop.
Marques and I were strongly aligned on the concept. We wanted a thriller with drama and strong, positive teen characters. But there would be clear messages as well: hacking is a positive activity, but can be abused. Good guys can use their hacking skills to protect people from the bad guys. Our concept for Hackerteen beautifully reflected my views and values about computing, policy, and free culture.
This is what I pitched when I returned home, and my managers got on board. Marques wrote the text with a colleague, Rodolfo Gobi, and hired an artist with a strong Manga influence and a Brazilian sense of dramatic color. I made a lot of edits. In particular, I chastised the authors for having an all-male team of hackers—and possibly they were all light-skinned as well. We added a girl and a dark-skinned character.
I also carefully checked the technical content, which dealt with intruders and an attempt to bring down the Domain Name System, to make sure it was accurate. I missed just one point: it was unclear how the bad guys had gotten access to the camera on one character’s computer in order to capture pictures of her naked and blackmail her. The mechanism was never explained, and turned out to be weak when I later asked Marques about it. Still the book was quite good both technically and as a story. It ended with the main character saving the Domain Name System but being framed and going to jail for another escapade.
I promoted this book like nothing else in my career. I mailed everybody I knew who had the slightest interest in free software or free culture. I mobilized O’Reilly around the project by holding contests in the Sebastopol and Cambridge offices. For the contest, I made a random-looking pattern of black and white squares and asked everybody to download it. I then created a number of similar-looking black and white patterns and asked some colleagues to help spread them all around the Sebastopol and Cambridge offices. Only one sheet exactly matched the pattern people were downloading, and I gave a prize to the people who first found the right sheet. Each office organized a party around the search.
In my mission to promote Hackerteen, I even set up a visit with Scholastic, the major publisher of children’s educational content with many books and magazines under their publishing umbrella (including one author you might have heard of named J.K. Rowling). I boldly went to their New York City office and pitched Hackerteen to a staff person. I showed her my favorite panel in the book: a somewhat climactic scene where the main character solves a difficult test to win placement on a team. The character and the test-giver gaze at a screen that erupts with glorious light, lending a religious aura to the scene.
The editor recognized my passion but explained that the Scholastic publishing schedule was very different from that of a computer book publisher. Scholastic plans some eighteen months in advance of release, and matches their releases to an academic schedule. There turned out to be no point of contact where O’Reilly’s publishing model could work with that of Scholastic.
Hackerteen failed to take off. The main reason was that we charged too much. Comic books tend to be very cheap, especially when children are the chief audience. The printed books are flimsy and the color is washed-out. Our book had to be beautiful to be effective, and after we invested the necessary amount to make that happen, we needed to charge $20 per copy. Furthermore, although our characters were teenaged, our colleagues said that the books appealed to preteens who were looking forward to becoming teenagers. Basically, neither I nor O’Reilly had any understanding of the comic book market, and we couldn’t break into it.
Marques invested a lot in promoting the book as well. Among other things, he set up a web site where the key technical elements of the book could be explained in simple language for preteens. I recruited many leading Internet activists to write text, and edited them.
I felt sorry for Marques, because once the book was released, he forged ahead and planned a second volume where the chief hacker’s girlfriend becomes a lawyer and gets him out of jail. The motives of the bad guys—who had some sympathetic elements—would be further revealed. This volume would have been even better, because we both really understood our message and the plot elements we wanted. Although I cautioned him that O’Reilly was not succeeding in selling Hackerteen Volume 1 and might not sign another book, he flew me down to São Paulo and then even further to Rio de Janeiro to meet his cartoonist.
The trip was exciting for many reasons. I got to visit again the author who had brought me to Brazil in the first place. Marques also took me to the Ipanema district of Rio, famous for Astrud Gilberto’s song, and to the museum of contemporary art in São Paulo. The museum happened at that time to have an exhibit about bossa nova, one of my favorite pop music styles, and I could read the Portuguese descriptive plaques. I thought back to this exhibit years later when I wrote my report about free software in Brazil.
Another pop music encounter was a dance that Marques and Gobi took me to. Gobi’s wife tried to teach me one of the dance steps, and someone used their cell phone to videotape my dancing. When I saw it, I recognized myself expertly carrying out the dance steps toward the left when everybody else was moving right, and dancing to the right when everybody else was moving left.
We also went to a book fair. Because the year was 2008, many books were being published in honor of the 40th anniversary of the revolutionary year 1968. It was interesting to see how publishers hooked onto the romance of that period, now so far removed in time. As an experiment, Marques placed a copy of Hackerteen on a shelf of some publisher, and we watched a child pick it up curiously and look through it.
But of course, the purpose of the visit all came to nothing. O’Reilly declined to produce the second volume. Marques expressed no hard feelings, and said he’d try to produce it himself in Brazil as a promotion for 4Linux. I have preserved several precious copies of the first and only volume, as well as some of the T-shirts O’Reilly printed to mark this ground-breaking attempt to bring our values and knowledge to young people.
Computer publishers are clench-jawed Gradgrinds. A book is supposed to teach something practical. It must present an identifiable goal for the editor’s routine spreadsheet of costs and earnings to pass approval. But what about a book that’s just supposed to show off beautiful things? How could that get published?
My acquisitions were already slumping in 2006 when a computer science professor at the University of Toronto, Greg Wilson, came to O’Reilly with a somewhat half-formed idea he called “Beautiful Code”. Although most people think of programs as functional, the programmers themselves—at least the top programmers—see them much differently. Like mathematicians and physicists, programmers can detect in good programs various intellectually satisfying concepts that they associate with beauty. The notion of beauty itself is beautifully amorphous, so if you ask any accomplished programmer to name a piece of beautiful code, they will come up with something different from other programmers. And this is what Wilson aimed to get: a set of marvelously varied examples of beautiful code, providing wonder on each page turn.
Although the field of computer publishing was teetering, at O’Reilly a curiosity for experimentation and a sense of wonder persisted. “Beautiful Code” was approved.
Wilson’s connections proved crucial. Experts whose fame would intimidate me or other O’Reilly personnel readily agreed to Wilson’s project. The more than 30 contributions we solicited ranged from a fresh look at sorting—one of the classics of programming—from Jon Bentley of Programming Pearls fame, to a bespoke software project that took text input from Stephen Hawking.
To reel in contributors, Wilson stipulated all royalties to be donated to Amnesty International. Because a book with more than 30 authors could not generate serious income for any single contributor, and because coordinating the distribution of royalties to so many contributors would be a logistical headache, Wilson’s decision was a brilliant solution. Many authors said that the donation to Amnesty International provided the main incentive for them to sign up.
True, a few recruits were dissuaded by our support for Amnesty International. One, expressing the narrow prejudice of nationalists everywhere, rejected the project because he claimed that AI was unfair to the government of Israel, and a couple others expressed timidity at being associated with the organization. But the only big problem came when our Beijing office chose to translate the best-seller into Mandarin. The office staff there confided to us that announcing the donation of royalties within the book would not stir up controversy, but that putting that information on the back cover could trigger strong criticism from the government. Amnesty International is hated and feared by the mainland Chinese authorities. On the other hand, although not deaf to these concerns, Wilson averred that leaving the information off the cover would constitute false advertising. Finally our Beijing office found a solution that involved referring to AI by just its initials instead of the whole name.
Even though many of the contributors were seasoned authors, my editing made a huge difference. Even Bentley expressed appreciation at my comments, which included an alteration to his code. Wilson and I handled input files in just about every format known to computing, from TEX to FrameMaker, and shepherded all the authors toward our deadline.
“Beautiful Code” was such a runaway success that O’Reilly created a whole web site devoted to it, with its own domain name beautifulcode.oreillynet.com, and launched a new series based on the idea of “beauty” in software. I myself contributed a story of achievement and protest at my former employer MASSCOMP to a book called “Beautiful Teams”. But none of the follow-on books sold well. “Beautiful Code” stood apart from all the rest in its fame.
The book took the prize in an important event called Jolt Awards, run by the well-known magazine called Dr. Dobb’s. (A few years, another book I edited, this one interviewing famous compiler developers, won another Jolt Award.) The day that I was scheduled to pick up the award, I heard that my mother died after a long bout with cancer at the age of 93. Nevertheless, I showed up for the reward and made a short speech praising the authors as the real winners. I cut my appearance short at the party that followed.
One of my last important contributions to book publishing took place in 2011 with another unique book proposed by Wilson: “Making Software”. Having spent much of his career in academic computer science, Wilson told me he was alarmed by the exaggerated claims often made about the efficacy of various popular or novel programming practices. He was frustrated by the assertions programmers made about the right way to do software engineering, usually with no evidence to back them up. When a new fad—or more often, a variation of some forgotten fad— receives attention, such as Agile programming or SCRUM, or when a new programming language takes hold, or even when a simple practice spreads such as pair programming (where one person watches over the shoulder of another and comments on their coding choices), people line up with testimonials or condemnations that are completely anecdotal. Entire books promote practices with no rigorous research, representing only the fervent testimony of a programmer who has found success.
Fields such as medicine and engineering have become fundamental to modern life precisely by eschewing such superficial impressions in favor of rigorous testing. Given the importance of software and the scarcity of proficient programmers, software needs some of that rigor.
Our book therefore aimed at a lofty goal suited to Wilson’s professional integrity. He gathered leading researchers in software engineering to write chapters for an anthology about what works and what doesn’t. Wilson and I collaborated as we had earlier on Beautiful Code, with him recruiting and me doing most of the editing.
“Making Software” offers a window into this research world in the form of literature reviews by recognized academic experts. The title conveys nothing of the book’s modus operandi, but we could think of nothing really apt. The book’s poor sales may be explained less by the title than by limits that its honesty placed on its claims. Software is nowhere near being a comprehensive and well-tested doctrine that can dictate practice. Our book could do little more than prove that many of popular assumptions are wrong and that we should not take much for granted. Software engineering research has turned up very few answers to the pressing questions programmers have: how to organize a team, what programming language to use, how to test, etc.
So our message was not appealing. Of course, most research in most fields remains inconclusive; that’s why confusion marked the world’s response to the COVID-19 pandemic for months. Too often, humanity’s plea for answers goes unmet.
The most urgent skill needed by a publisher is not editing, but networking. We have to stay in constant touch with the people pushing technology forward. To publishers, “cutting edge” feels as visceral as the actual touch of a blade. The people we cultivated dropped insights into our conversations like nickels into a beggar’s cup, and those insights might be in some area quite unrelated to the expertise they gave us for their books or conference presentations.
For instance, I started hearing around 2000 that very geeky developers were buying Apple Macintosh desktop and laptop systems. This intrigued me, because at that time, the Mac was popular among designers and similar artistic professionals, but hard-core computer professionals passed them by, finding them deficient in the tools the professionals needed. Among my crowd, professionals used GNU/Linux systems. What led to the sudden enthusiasm for Macintoshes?
I’m sure the shift came because Apple based its new system (MacOS X) on a Unix-compatible kernel. They took some oddball version of BSD and altered it further to make a version of Unix they called Darwin. They even released this kernel as free software. Suddenly, the powerful developer tools that had long dominated the field, thanks to Unix workstations and then GNU/Linux, were a quick port to the Macintosh. My company also shifted to Macs, although I didn’t get one until I found my GNU/Linux laptops lacking in the functions I needed. At the Open Source Convention, where we still provided a large room full of desktop systems with Ethernet connections for people without laptops, I saw one year that these were all Macs.
In my daily blog posting, I noted the incongruity of setting up proprietary desktop systems at an open source conference. This was nearly the only time I got in trouble for blogging, after doing it weekly for many years. A conference manager noticed the observation (not really a complaint) in my posting and contacted me to explain that the room full of desktop systems were part of Apple’s generous sponsorship for the conference, and that we must not anger them. The sentence got stripped from the posting.
Well, tact is not my strong point. I write notable articles because I notice things, and then note them vigorously in my writing. In this case, I admit, I violated not only corporate financial calculations but basic etiquette.
Only one other time, at least five years later, did my penchant for broad observation get me in trouble. Tim O’Reilly and the company had been following reform efforts to insert data processing and digital communications into government, and decided to take advantage of the gathering movement toward transparency and scientific government.
Carl Malamud, a technologist who came very early to the call for opening government data, spoke at our Cambridge office back in the 1990s. Others came later with digital solutions to common problems, such as the FixMyStreet app. This app did nothing more momentous than making it easy for citizens to report potholes or trash, but it offered the chance for local governments to build trust among their residents.
Efforts like these cascaded. I was well poised to join O’Reilly’s efforts, having met Beth Noveck in 2005 and followed her work on Peer to Patent into a general fascination with open, digital government.
Tim himself wanted to take a leading role, and built tight relationships with technically sophisticated leaders in the Obama White House (where Noveck also spent a couple years). As he spoke and held his usual rounds among leaders in the field, the company struck up partnerships and sponsorships. One of the new organizations that made a splash was Code for America, which took a leaf from the notebook of Teach for America. Young people sponsored by Code for America would work with local governments to solve problems through the development and dissemination of apps. The founder of Code for America, Jennifer Pahlka, took on a number of leading roles at agencies and married Tim.
The term Government 2.0 came to cover this combination of public engagement, transparency, and better living through data crunching. I believe Tim himself invented the term, although I cannot be sure. Tim had definitely invented the term Web 2.0, which helped observers of business and technology in the 2000 decade understand the burgeoning role of private contributors to the Internet commons.
The 2.0 moniker, which came from the version numbers bandied by computer developers to flaunt important improvements in software or hardware, had been applied metaphorically in other settings to say, “Look up, folks! Things are different now. Out with the old, in with the new.” This kind of fad, where catchy terms skip from one context to another, is common. In the 2010s, after some programming teams decided to combine development and operations (deployment) under the catchy term DevOps, other people felt impelled to add the suffix Ops to everything, whether it made sense or not.
After Web 2.0, it would be easy to slap 2.0 on other everyday terms. When our company got interested in health IT, we were non-plussed to learn that Health 2.0 had already been trademarked by another organization. It was a strong and innovative group, led by Matthew Holt and Dr. Indu Subaiya. I attended many of their conferences and wrote for their blog. O’Reilly negotiated to buy their company, but couldn’t come to terms both sides liked. And a couple years later, O’Reilly was out of health IT.
Government 2.0 didn’t last long for us either. Behind all the bright lights and blogging, our business strategy was to sell our books and other offerings to government agencies. But they weren’t buying. Many people would express interest in the themes of Government 2.0 but couldn’t find forty bucks in the budget to buy a book about it.
This was truly a major company focus for a couple years. In the thick of our most earnest efforts, I wrote a blog posting where I said that the term Government 2.0 was misapplied. Having just seen a shocking exhibit of frescos from ancient Assyria, I suggested that empire was the first government and that the Greek democratic innovations were the real Government 2.0. What was my point in churning up all this ancient history? That our efforts at rationalism and transparency were just a continuation of the democratic ideal, and a modern way to bring it closer.
This time I got email quickly from Tim himself. He explained that he and the company had invested a lot of money and branding effort in the term “Government 2.0” and couldn’t allow my posting to undermine it. Apparently, Tim had access (or asked our web team for it) to all postings on the O’Reilly blog, because he altered my posting to smooth out the reference to the Assyrians and Greeks. I admit that Tim, having a degree in classics from Harvard, might know more about the history of Greek democracy than I do.
So those were the only two times, during a couple dozen years over which I wrote hundreds of postings for their blog, when O’Reilly managers took out observations because they went against the company’s business interests. I don’t mind that at all. I realize that hard-hitting and insightful reportage is like sending a shopping cart full of merchandise careening down the aisle of a retail store. If someone has put an item in your way without telling you, it can get hit.
But what about the correctness of my writing, in my own judgment? Do I regret anything I put in my blog postings, journal articles, or conference presentations? Yes, three times. One concerns a misunderstanding of statistical techniques, which I describe elsewhere. Another was an article I hastily wrote about a particular patent dispute without adequate research; I received a lot of criticism and took the article down. The third one concerns a policy recommendation I put in an early article of mine about encryption, which I’ll describe here.
Digital encryption has scared the security forces of all countries since Whitfield Diffie and Martin Hellman figured out how to do it right in the 1970s (and their system has never been proven ineffective—only particular implementations can be broken). Every few years, some security agency suggests that encryption be banned entirely, or weakened in some complicated way that the agency thinks they can push through because it’s too confusing to alarm the public and draw criticism. Sometimes they claim that criminals won’t get in between the private sender and recipient, but only the beneficent agencies as “trusted third parties”.
The agencies have always failed to persuade the public that we’re safe with their hacks to the system, because people who really understand cryptography—people who devote their lives to understanding the algorithms and associated risks—are naturally those who want cryptography to work. They swarm over every proposal and demolish the pretense that it can preserve privacy for the average user while giving law enforcement the access it wants.
Back in the 1970s and 1980s, the situation was already contentious, with the U.S. government imposing absurd export restrictions on software that people outside the U.S. could use to protect communications. Did our government think that no one outside the U.S. could figure out how to implement encryption software? The only thing accomplished by those regulations (which eventually were successfully challenged on free speech grounds) was to give headaches to people working across borders on free software. Similarly useless restrictions were later placed on software that could read copyrighted media, such as digital movies that were “protected” with weak key systems.
The debates have become even more urgent with the advent, on the one hand, of digital commerce and online banking where everyone needs strong protection, and on the other hand, of far-flung terrorist networks who have become expert at bringing new recruits onto hidden communication channels.
This is where my mistake comes in. Amidst all this brouhaha, in the mid-1990s, I was asked by some UK-based lawyers to join them in an article criticizing attempts to control encryption. I was quite new to policy issues, and unused to thinking through ramifications of policy proposals. I trusted these lawyers, who had admirable values and impulses. It was also a rare opportunity for me to put forward my name as a co-author in a research journal. So in June 1997, the Journal of Information, Law, and Technology published our article with the righteous title “Can the Trusted Third Parties be Trusted? A Critique of the Recent UK Proposals.”
The critique was standard stuff for Internet activists, and remains valid. The problem was that some author had thought up a solution they thought was cool. In their proposal, everybody using a key would have to submit it to a secure archive, where governments could subpoena encryption keys from suspects in the same way they get search warrants for homes and businesses.
Unfortunately, this exact solution emerged later in a U.S. government proposal known as the Clipper Chip. Even though I doubt that those policy-makers had read our article in a minor U.K. journal, I am embarrassed that I went along with this solution. The Clipper Chip proposal was instantly blasted to smithereens by the crypto community. In retrospect, not only was it woefully insecure and intrusive, but I can’t imagine the logistics of such a proposal as I generate new encryption keys two or three times a week for various web sites. The worthlessness and absurdity of this “key escrow” idea does not protect it from being regularly raised in fresh settings by new policy-makers.
This excursion into Government 2.0, Health 2.0, and encryption was, I hope, interesting because they covered some crucial historical moments, but I got off topic. Remember what this section of the memoir started off talking about? Before I got into the origins of MacOS X, I was discussing the role of networking.
Among the many authors and fellow editors whom I now count as friends, I formed an unusual relationship based on our shared appreciation for literature with Dinesh G Dutt. This network expert started out at O’Reilly writing two reports on old networking protocols that have emerged with new importance in modern data centers. He then proposed a full book on how to design and configure a data center. I was honored to be assigned to all these projects. Dutt was a central figure in networking; he had actually created one of the protocols he was documenting for us.
After he expressed his dream of becoming a strong writer, we started to discuss literature and poetry. I wrote a poem using the themes and terms I found in his book, and dedicated it to him. He in turn liked the poem so much that he asked for permission to include it in the book, and the managing editor agreed. This became the first O’Reilly book to contain an original poem.
From stories such as this, it will be clear that the friendships I formed with authors were strong and productive. I was repelled by the way some publishers would talk about having a “stable of writers”. My authors were human beings to me, and our work embodied many aspects of humanity.
One can well ask where the women are in all these creative encounters with technology. O’Reilly certainly found fewer female authors than male ones, but we published several significant women. One author with whom I worked was Elecia White, who joined a series we were publishing on embedded systems with a very personal approach. Her book Making Embedded Systems seemed to me exactly what an entrant to the field would want after getting a degree in electrical engineering or computer science. White bridged the gap between book knowledge and job performance by presenting a grab-bag of important skills such as reading schematics and balancing resource constraints with requirements.
White’s creativity was a risky in the marketplace, just as for a novelist whose book that doesn’t fit neatly into some genre such as mysteries or historical fiction. Publishers and booksellers don’t know how to advertise, shelve, and otherwise promote something that falls outside hidebound categories. White’s book didn’t stick to the traditional scope of embedded systems. In contrast, most of our embedded series books—which I had proposed and edited—concerned Linux or its derivative Android, and stayed within their technical boundaries.
In addition, White’s book came out in 2011 as our embedded series was in decline. Even while home devices and the Internet of Things were rendering embedded systems increasingly a part of our lives, readers were turning away from tried-and-true titles we offered in the embedded space, and often a highly popular book would fail in its second edition.
So sales of her book started out slow, and I was disappointed. However, in her review of this memoir, White tells me that sales have been steady, probably be because her book covered useful, general topics and didn’t become obsolete like so many technical books. In the ensuing nine years, she has enjoyed some $30,000 to $35,000 in royalties.
The topic of this section allows me to stand back and sweep my eyes across my career to establish the context in which I came to O’Reilly. When I accepted the job, I thought I was leaving the computer industry proper and taking on a diminished role as an outside observer from the publishing industry. I was unfairly denigrating the company and my own capabilities, because O’Reilly uniquely participated in the communities we wrote about. But now I understand that my career move represented a more significant shift, moving with the computer industry as a whole. Coming to O’Reilly in 1992 allowed me to escape the failed computer development model of the Open Software Foundation (OSF), which I described in another chapter, and to take on a winning model.
OSF itself looked like a way to escape the declining environment for high-performance computers at MASSCOMP. My former boss from that company, Steve Talbott, tried to recruit me for O’Reilly at that time. However, I decided instead to try out Hitachi, one of the world’s major corporations that was attempting the kind of hopeless project to which I myself am partial. One of Hitachi’s less notable divisions was sputtering along by selling a mainframe computer compatible with IBM’s classic line. By joining the Open Software Foundation and adopting its Digital Computing Environment (DCE), the division hoped to drink deep from the fertile Unix watershed. DCE came as a new ripple in the decades-long stream of contortions by the computer field to build powerful networks from systems that had been designed since the 1940s to be stand-alone processors.
OSF was located in Cambridge, Massachusetts, so it seemed reasonable to Hitachi to open a small office in the suburbs of Boston where they would implement DCE. I lasted there for eighteen months; perhaps another year or two was needed to kill the project entirely at Hitachi while OSF itself imploded.
I am happy, though, for the time I spent at Hitachi. The company and its managers were compassionate and intelligent, and my time there yielded many lessons that steadied my path through the future years.
At Hitachi, I had access to a lot of classic Unix source code. I became used to consulting programming code for the products I was documenting. This proved useful at O’Reilly, particularly for the books I edited on the Linux kernel. I soon overcame my earlier intimidation when facing the high priests of the software world, having seen some incredibly ugly and convoluted contributions that were written by the original BSD developers and were now powering computers and the Internet everywhere.
Also at Hitachi, the small writing team quickly saw my facility at learning and developing tools, so I became the go-to person for all kinds of technical problems in building documents using the documentation utilities that OSF provided us. These were, not surprisingly, as buggy as the rest of the software our development team was expected to bring to market.
Hitachi management recognized the importance of training American staff in the culture and mannerisms of Japanese companies. The philosophy and practices I picked up during formal and informal instruction made me more thoughtful and tolerant. I believe my short stay made me more patient with everyone’s differences and more able to adapt to a global culture and economy, which the Internet and the Washington consensus (remember that?) were quickly bringing us.
Not long after I started at O’Reilly, DCE came back into my life. This time I did not mind. The work was planned around quality and around meeting users’ needs, and hence proved much more rewarding. I was given responsibility for a new series desired by Digital Equipment Corporation, one of the most important New England computer manufacturers, and both a contributor to and seller of DCE.
DEC and DCE—these went together in my life now. I’ll refer to DEC as Digital to ensure that abbreviations don’t get confusing. I knew the company quite well, as my wife Judy had worked just a few years before at the famous mill where Digital began in the town of Maynard, Massachusetts, and I had many friends within the company who kept me updated on their strategy, offerings, and stumblings. Digital facilities sprawled across Eastern Massachusetts and Southern New Hampshire at the time of their partnership with O’Reilly.
I had long admired the documentation Digital produced for their largest product offering, the VAX/VMS series. The manuals were written not only simply and clearly but with panache. Even more important, the books showed the marks of being written by people who actually used the systems, and loved them.
Now Digital wanted to furnish writers to produce books on DCE for publication by O’Reilly. These writers were happy to accept my editorial guidance. We got along great, feeling like a single cross-company team. Digital was known for matrix management, where people officially cross organizational boundaries (such as between development, testing, documentation, and marketing) to build products collaboratively. O’Reilly’s informal modes of collaboration fit in perfectly. Results were high quality.
I drove regularly to New Hampshire to meet my Digital colleagues. I got very friendly with authors Ward Rosenberry and John Shirley. Rosenberry brought his family over to my house and our kids played together. I also basked in the generous aura of Frank Willison, manager of the Digital team that was writing our books.
I believe that our first DCE book included, in its preface, the author’s acknowledgment of support from his gay partner. Demonstrating a precaution still appropriate in the mid-1990s, I double-checked to make sure the author understood that this book was going to be sold publicly.
DCE failed because it sought too hard for the Holy Grail of computing, which is to make multiple systems in different places feel like a single system. Some progress toward that goal had been made in the 1980s and 1990s through networked file systems, which included Sun Microsystem’s NFS and a competing system called SMB that Microsoft licensed and that proved more functional than NFS in the long run. Another important feature of distributed computing is the distributed register, which also exists in bifurcated form: LDAP in the free software and Unix world, and Active Directory (based on LDAP and somewhat compatible with it) in the Microsoft world. These allow a single source of truth for important information about employees and departments throughout an organization. I edited a book on Active Directory during my sabbatical from Linux and free software in the mid-1990s.
Another monster called CORBA (the creators couldn’t even force the title into an easy-to-pronounce acronym) attempted to extend the darling programming model of the 1990s, object-oriented programming, across network boundaries. CORBA reminded me of OSF products, because all the big companies clustered around the initiative but proved incapable of true collaboration. As every company’s implementation differed and they never worked together—integration being of course the whole point of CORBA—the project kept adding higher and higher levels of the putative standard, each layer striving to harmonize the incompatibilities of the layer below and, in Sisyphean fashion, creating a hotbed for new incompatibilities.
And yet the industry hype around CORBA was so huge that O’Reilly set its editors to the task of signing a book on CORBA. Existing books were as indecipherable and gangly as the product itself, and I saw no way to treat the subject in a comprehensible manner. It also moved too fast to tie down in book form—everything seemed to be outdated as soon as it was published. Eventually CORBA shriveled up before we could find an author. The project demonstrated several traits that turned up in later projects and provided great dilemmas to publishers. In particular, the pace of development in computing picked up a great deal in the 1990s, so books were not keeping up and the professional computing public was coming to depend on the richer and richer online information sources.
The residual embers of DCE’s impact remained, ironically, in Microsoft’s products. I know of just one useful invention that DCE promoted and that remained important to computing: the UUID. This was meant to solve a fundamental requirement of distributed computing: how to distinguish people, computers, and other items with identifiers that could be generated simultaneously on thousands of different computers but be reliably assumed to be unique. Apollo Computer, which was another towering success of 1980s computing for a brief time, invented the UUID. Through DCE, it became seen as a useful feature now implemented in programming libraries everywhere and employed through computing for sundry purposes.
As one can imagine, knowing the train wreck that was DCE, our books did not sell well. But along the way O’Reilly and Digital worked with Microsoft, which had suddenly decided to boost its appeal in the Unix market by supporting DCE. Their goal was not to become a Unix shop, but to make it easy to tie Unix computers to their Windows systems, which were now running a new operating system called Windows NT that they had built from the bottom up to support the familiar (although every-morphing) Windows interfaces. Rosenberry wrote a book on Microsoft’s implementation, with me as editor. Microsoft maintained their interest in O’Reilly and commissioned us to produce their Microsoft Press books for online and print distribution many years later, in recognition that our platform and production process were tops in the industry.
The company Digital was going downhill at the time they were working with us on DCE books. Just as I jumped ship at Hitachi, Frank Willison saw the writing on the wall at Digital and took a job as manager of the editorial team at O’Reilly. He played an important role there for years.