--- title: "Supply Chain Roulette" date: 2026-02-14 description: "I watched a supply chain attack unfold in real-time. Here's what I learned about trusting third-party code." tags: ["security","supply-chain","zero-trust","incident-response"] readingTime: "10 min read" url: https://alexmoening.com/dev-thoughts/supply-chain-roulette.html markdownUrl: https://alexmoening.com/dev-thoughts/supply-chain-roulette.md --- # Supply Chain Roulette [← Back to /dev/thoughts](/dev-thoughts/)

Every <script src="https://cdn..."> tag is a trust relationship. When that CDN gets compromised, so do you.

### The Convenience Trap

CDN embeds trade security for convenience. Most developers don't think twice about it.

It's the path of least resistance. Need p5.js? Grab it from cdnjs. Need a utility library? unpkg has you covered. The pattern is everywhere:

<!-- The path of least resistance -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.7.0/p5.min.js"></script>
<script src="https://unpkg.com/some-lib@latest/dist/bundle.js"></script>

Convenient. Fast to implement. And potentially catastrophic. ### When CDNs Go Wrong

One compromised CDN can inject malicious code into millions of sites simultaneously.

In February 2024, polyfill.io was acquired by a Chinese company called Funnull. Within months, the service that millions of websites trusted to provide browser compatibility code began injecting malicious JavaScript. The attack was sophisticated: malicious code only activated for specific user agents, making detection difficult. By the time security researchers caught it, **over 100,000 websites** were compromised. This wasn't an isolated incident. Magecart attacks, which target e-commerce sites through compromised third-party scripts, have affected over 2 million websites since 2015. The attack vector is simple: compromise a CDN or package registry, and you instantly have code execution on every site that embeds from that source. No need to hack each site individually. One compromised dependency, millions of victims. ### The Night the Stream Went Dark

I watched a supply chain attack unfold in real-time at Akamai. That hour on the Bravo Bridge changed how I think about trust.

I saw this play out firsthand during my time at Akamai, when I worked as a Solutions Engineer supporting video streaming customers. One of our customers was Ooyala, a video platform that powered streaming for major media companies. Their architecture relied on a JavaScript player that customers embedded on their sites--and that player pulled in additional assets from Ooyala's CDN at runtime. Configuration files, UI components, analytics scripts. Standard architecture for the era. Then came the spear phishing email. Someone inside Ooyala--an engineer or administrator with elevated access--fell for a targeted attack. It's worth saying: this can happen to anyone. Spear phishing is sophisticated, personal, and designed to bypass your natural skepticism. No fault on the user. The fault was in the system that allowed a single compromised credential to cause this much damage. The attackers got credentials. Not to the code repository, but to something arguably worse: the CDN configuration. They didn't need to change the player code. They just needed to change what the player *fetched*. The target was a major news broadcaster--a right-leaning network known for its hosts' strong political views. When viewers tuned into their live stream that evening, instead of news, they saw a political message. Hacktivism, delivered through a trusted embed. Here's the thing: we dodged a bullet. The attack was surgically targeted--it only activated for requests with a referer from that specific broadcaster's domain. Every other Ooyala customer was unaffected. If the attacker had been truly malicious, they could have compromised every site using Ooyala's player. Credential harvesting. Cryptojacking. Keyloggers. Instead, they wanted to make a political statement to a specific audience. The limited scope is what convinced us this was hacktivism, not a criminal operation. #### The Bravo Bridge The phone rang. It might have been the first time I got paged into Akamai's "Bravo Bridge"--the company's major incident response channel. Akamai was known for its incident management practices, and that night I saw why. The bridge followed a formal structure--roles you'll find in any mature [incident response framework](https://www.atlassian.com/incident-management/incident-response/roles-responsibilities):

Role	Responsibility
`Incident Commander`	Central authority during crisis—coordinates the response, makes decisions, keeps things moving
`Communications Lead`	Manages stakeholder updates, shields the technical team from external noise
`Operations Lead`	Directs technical responders, coordinates diagnostic and remediation efforts
`Scribe`	Documents everything in real-time—maintains timeline, captures decisions for the postmortem
`Technical Responders`	Subject matter experts from each affected system, hands on keyboards

Technical incident management is a discipline unto itself. That night, I gained a deep appreciation for how structure and clear roles can cut through chaos. But even with that structure, there was nothing we could do but wait. #### The Longest Hour Rollback was initiated within minutes of identifying the attack. But CDN configuration propagation was slow back then--changes had to reach edge servers worldwide. Even expedited, rollback took about an hour. That hour dragged on forever. Today, CDN configs propagate in seconds. Back then, we watched the clock while a political message played on a major news network's live stream. The broadcaster didn't know what hit them. From their perspective, nothing had changed on their end. Their site was exactly as they'd left it. The attack came through the supply chain--through the embedded player they trusted, fetching assets they never audited. #### The Aftermath The relationship with the broadcaster was damaged. Trust, once broken, is hard to rebuild. But something unexpected happened with Ooyala. The incident actually strengthened the bond between Akamai's account team and Ooyala's technical team. They were impressed by our response during a difficult incident and wanted to help us understand what went wrong from their perspective. That collaboration led to hard conversations about what needed to change. The postmortem was sobering:

Lesson	Action
`MFA everywhere`	Critical infrastructure requires multi-factor authentication. One password shouldn't unlock production.
`Lock the front door`	CDN configs are your entry point—treat them like the keys to the kingdom. Double-lock them.
`Zero trust`	Even authorized users must prove who they are for sensitive operations. Assume credentials can be compromised.
`Least privilege`	The engineer didn't need write access to production CDN configs. Neither does anyone else by default.
`Revoke and rotate`	Compromised credentials trigger immediate key rotation across all systems.
`IP allowlisting`	Lock down the CDN control channel to authorized IPs only. More brittle, but security is about trade-offs.

That last one became a product feature. After the incident, we instrumented IP allowlisting for CDN configuration APIs. Customers could now restrict who could push changes to their delivery configs--only requests from their authorized IPs would be accepted. Yes, it made things more brittle. Any config change had to come from a specific IP owned by the customer. But that brittleness was the point. It raised the bar for attackers significantly. This happened early in my Akamai career, before "zero trust" was a buzzword. But the lesson was clear: trust no single authentication factor, trust no single access control, and definitely don't trust that someone "authorized" is actually who they say they are. CDN configurations are the front door to your entire delivery architecture. You wouldn't protect your front door with just one lock. ### Securing the Pipeline

The real threat isn't just external attackers—it's anyone who remembers a pipeline password after they leave.

To be clear: none of these modern protections existed when the Ooyala incident happened. Multi-factor authentication for infrastructure was rare. Zero trust was an academic concept. We did the best we could with what we had. Today, you have options. The average insider threat incident costs **$17 million** (2025 data). The most dangerous scenario isn't a sophisticated external attacker--it's an admin who leaves your company and still remembers how to push to your CDN configuration pipeline. Modern infrastructure provides multiple signals to verify that a request is legitimate:

Method	How It Works
`Mutual TLS (mTLS)`	Both client and server authenticate with certificates
`OIDC federation`	CI/CD exchanges a token for temporary credentials—no long-lived keys
`IP allowlisting`	Restrict access to known source IPs
`Adaptive MFA`	Adjust auth requirements based on real-time risk signals

The key insight: **layer these methods**. A request to update your CDN configuration should require multiple signals--not just a valid API key, but also: originating from an allowed IP, using a short-lived token from OIDC federation, with mTLS verifying the runner's identity. ### The Self-Hosting Alternative

Verify once at build time, serve a known-good copy forever.

For the code you embed on your sites, the solution is straightforward: host your dependencies yourself. Download the libraries you need, audit them, and serve them from your own infrastructure.

<!-- Before: Trust the CDN -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.7.0/p5.min.js"></script>

<!-- After: Trust yourself -->
<script src="/lib/p5.min.js"></script>

This shifts the trust model. Instead of trusting that every request to a third-party CDN returns the same safe code, you verify once at build time and serve a known-good copy. But self-hosting is only half the solution. You still need to verify what you're hosting. That's where a security pipeline comes in--hash verification, pattern scanning, and human review before anything reaches production. I've written a companion piece on [implementing a security audit pipeline](auditing-your-dependencies.html), including how to use LLMs for code review (and why you shouldn't trust them too much). ### The Lesson That incident shaped how I think about third-party embeds to this day. When you embed a script, you're not just trusting the code you see--you're trusting everyone who can change what that script loads, every configuration it reads, every asset it fetches. The attack surface extends far beyond the initial `