Tech debt is eating the world

Web infrastructure begets inequality, especially for tech nonprofits

Thomas Struth, Neutral Buoyancy Lab, JSC, Houston, 2017

It’s a strange time to resurrect this newsletter, but the truth is that I need to write to make sense of what’s going on. I keep coming back to web infrastructure: who owns it and who benefits from it?

Marx will tell you this question has always been relevant, but I wonder if it’s even more relevant right now, at this time when the internet and its infrastructure is increasingly opaque, abstracted away. You’ve got the internet, and then the services that run atop it, and then the apps and things we use to access those services — three layers, and those are the obvious ones. What’s obscured is how these services work and who really powers them — not to mention the role of our own little labor in this growing apparatus.

My favorite illustration of this is the Anatomy of AI, which looks under the hood of the Amazon Echo, revealing our own unpaid labor at the core, encased by big tech infrastructure, and surrounded by a web of proprietary process for extraction, refinement, and manufacturing. The complexity eludes comprehension.

Now imagine the infrastructure that powers a mutual aid website. It’s tiny and brittle and borrowed. The labor is voluntary, the code is stale. Underlying providers, like AWS or GoDaddy, can pull the plug at any point. That’s the case for so many tech nonprofits and mission-driven companies. And that’s what I’m interested in: how web infrastructure begets inequality.

Today I want to look at that spectrum spanning the the brittle infrastructure of mission-driven tech to the robust architecture of authoritarianism. Agenda below:

• What is tech debt?
• A case study on government infrastructure and tech debt: Healthcare.Gov and COVID test distribution
• A framework for nonprofits on using tech debt strategically
• The role of infrastructure in the Russia-Ukraine conflict

Tech debt and mission-driven tech

Happy families are all alike; each unhappy family is unhappy in its own way

— Tolstoy, on the idiosyncratic shitshow nature of workplaces

What is “tech debt” and why does it matter? Tech debt is like credit card debt — it’s an IOU. You cut a corner today at the expense of tomorrow. Maybe you built a really quick prototype and ended up pushing that live. Or maybe you used Shopify for your online store instead of building a custom payflow. Or, if you’re a nonprofit, maybe you jury-rigged your Salesforce application to track donations and send monthly emails and store customer data — all sorts of ungodly things.

Reforge put out a very good piece on tech debt that is worth reading. In it, Keya Patel very smartly analyzes tech debt and determines that, like real debt, it can and should be use strategically. You can take on debt deliberately, and come up with plans to pay it down. Here are a few different ways she thinks about it:

By classifying debt, you can get smart about how you use and lose it; you can prioritize it against other items on your product roadmap:

We’ll come back to this piece in a minute.

You don’t have to be a tech company to accumulate tech debt. Every organization I’ve worked at has accumulated tech debt in the process of adopting or building digital tools. The truth is tech debt is inevitable, and especially so for resource-constrained mission-driven organizations.

It’s no coincidence, either. The whole “tech for good” movement came about because of the widespread availability of free and cheap digital tools. It’s easier than ever for a nonprofit to go digital and expand its reach and impact. But adopting those tools has a cost. The deeper an org gets into implementation, the more they’ll need to customize to their tech or add in adjacent services. Soon they may require custom code and other engineering. And the larger their tech stack grows, the more risk they assume in tech debt.

On top of that, tech nonprofits are especially vulnerable to tech debt because of the labor market. These days, companies are less loyal to their employees, and vice versa — a failed prisoner’s dilemma. High turnover means that the people who start large software projects are not the ones who see them through implementation and launch. And so, the few engineers an org does have must spend time retreading and detangling past decisions and old code. That doesn’t make for particularly inspiring work, so talented engineers who were once interested in the mission-driven work leave for greener pastures (and much higher pay at Big Tech firms) — a vicious cycle. It’s tough out here for nonprofits.

A brief case study on tech debt and government

Government’s a great example of how tech debt hurts deep. Remember the fiasco that was the launch of healthcare.gov in 2013? Whatever corners they cut to speed up the launch, continue to plague the platform today. Even simple code changes can take months to make.

Fast forward to the recent COVID test rollout, which the Biden Administration seemed to handle well. If you ordered your Biden swabs, you probably noticed that it was shockingly easy. Unlike so much government tech we’re used to, it worked? You could sign onto USPS.com and add your tests to a cart and check out within minutes:

The team charged with this rollout did a really smart thing: rather than build a new web app, they piggybacked onto existing functionality within the USPS website. They used a templated page that looked a lot like a change-of-address form. Sure, it felt a little kludgy to “order” your COVID tests and have to “check out” — but it meant they could existing workflows that didn’t require new code. It was a big success:

It’s an interesting case study in terms of tech debt, and I’m curious to see how the infrastructure holds up in coming years. On one hand, the team used a dependable, readymade tool. On the other, they didn’t build out a new site for the express of purpose of distributing tests. With the current product, you can’t edit the number of tests, or specify the type of test, or even note an apartment number. That worked fine this time around. It was a calculated risk. They’re betting that what they have now will be able to serve them in the future, and/or that they can buy themselves enough time to build a custom solution for the next round of Biden swabs. Time will tell if it was the right long-term decision, though, again, it seemed like the smart play for the short term. (Never mind that Biden’s tests took weeks to roll out, arriving after Omicron peaked.) All mission-driven organizations must make these decisions as they digitize, and the consequences can be far-reaching.

Advice for nonprofits navigating software development and tech debt

I want to reiterate that tech debt is ubiquitous. I’ve seen it and worked on it at every organization I’ve been employed. The difference for nonprofits is the lack of in-house expertise to identify, classify, and strategize around it.

But all is not lost. The best solutions are free. Like so many other workplace issues, tech debt benefits from honest discussion. A lot of what you can do doesn’t require technical tools — just alignment and prioritization. Here’s how you can address as you build new software:

• Be intentional about every infrastructure decision. You have to recognize that every decision around your tech stack is consequential. Every time you add in a new service or set out to build a new application, stop to have a conversation. Infrastructure here is defined very broadly as any digital tool you may use in the course of business.
• Document those decisions: As you evaluate a new technology, document your decisions. List out pros and cons, identify alternatives, explain the rationale. This can be as simple as a one-page memo, or a more formal RFC like the one outlined here (“RFC” = request for comment).
• If you see something, say something. As the project plays out, flag any changes to your tech stack or decisions that may have lasting consequences.
• Question sunk costs. There is a human tendency to keep things moving. We figure that because we’re already made lots of investments into some endeavor that we have to see it through (aka the “sunk cost fallacy”). This is exacerbated with software projects, which are often too complex for a single individual to grasp. All the more important to pump the brakes. I can think of a few situations from my career where I wish I’d paused to question tech projects I’d inherited.
• Analyze your tech debt. Finally, use the Reforge matrix to classify your tech debt. Call out the work, and scope it, and add it to your roadmaps. If it’s important enough to work on, it’s important enough to report out like you would a user- or customer-facing feature.

This is all easier said than done, of course. Questioning an old project or ongoing endeavor can be delicate. When everything’s a priority and there’s so much to do and you’re really short-staffed, it can feel like there’s no time to pause. Make the time, thank yourself later.

Web infrastructure and the war on Ukraine

Indeed, infrastructure’s been crucial to the new kind of war unfolding in Ukraine’s physical and digital territories. I wanted to spend more time on this this week but I went too far down the infra hole and I’m out of steam. If you’re interested, start with Jon Favreau’s conversation with Kara Swisher, where she details how this is our first “Very Online War”. A few of my own observations from around the web:

• Information is the foundation for this conflict. Russia’s airwaves are filled with propaganda, and not much else. As soon as the war started, Putin deployed a digital blockade against alternatives points of view. The result is a chilling precedent: “Russia’s cleaving off is a defeat for the once-held Western belief that the internet is a tool for democracy that would lead authoritarian countries to open.” Masha Gessen and Ezra Klein dive deeper into this theme in an excellent episode of Ezra’s podcast.
• The personal is political and all the world’s a stage. President Zelensky — who played the Ukrainian president on TV before assuming that role — has used social media to masterful effect, controlling the narrative on the global stage. (Life continues to imitate art and life and art and so on… what is this world?)
• Activists across borders. Despite the blockade, Western activists have used services like Tinder, Google Maps, and Telegram to speak with Russians and counter propaganda, via Wired.
• News organizations are evolving. The New York Times just this morning announced they’re starting up a channel on Telegram to get facts to people on the ground in Russia and Ukraine.
• Tech companies are owning up. Platforms and tech companies are finally acknowledging their role in the new world order; they’re taking sides and taking action, largely against Russia. That’s great, in this instance. But it’s a strange precedent. It sheds light on what we all already know to be true: companies are more powerful geopolitical actors than many states. My hope is it will be harder for companies to claim ignorance and plausible deniability for their actions and externalities. A few examples of what companies have done against Russia, via Protocol:

Feb. 28: Roku removed RT from its channel store.

Feb. 28: TikTok banned state-backed news organizations from accessing their accounts.

March 1: Meta restricted access to Russian state-run news outlets RT and Sputnik in the EU.

March 2: Reddit banned links to state-supported Russian media outlets across its site.

March 3: Spotify removed all content from Sputnik and RT.

• But who will check Big Tech? The Electronic Frontier Foundation urges platforms not to change rules or to bow to public pressures. Wartime is a bad time to set new precedents, they argue, and they’re probably right.
• All streaming on the western front. The Open Source Intelligence (OSINT) Community is uncovering truth and mistruth in real-time. News outlets, flooded with video from social media, have to use increasingly sophisticated methods to uphold their veracity. CNN uses cross-references weather, geotags, satellite imagery, and metadata.

It’s a lot all at once, and it’s too soon to draw any real conclusions about the changing nature of power and politics. But what’s clear is that web infrastructure underpins and overwrites it all in this brave new world.

Links from around the web

• My brilliant friend Aaron Reiss takes a look at Chinatown’s infrastructure, and the rich history behind Chinese street signs.
• Subscribe to Astra, the new international magazine of literature, brought to you by another brilliant friend, Nadja Spiegelman.
• The New Yorker takes a look at Google Search and what it isn’t showing you.
• Substack moves to introduce its own app in a bid to circumvent email platforms like Google and Gmail — who ultimately determine the fate for Substack’s creators and their email content
• “Web 3 and the ‘For Good’ Trap.” Decentralized tech does not equal decentralized power. Nice piece in the Stanford Social Innovation Review.
• “Sober Questioning,” a great new column in NYMag on the highs and lows of sober living.
• I’m also in like with a new running newsletter I discovered on NYT.

Until next time, be best. — XML