Tag: gcdigital

“Build vs buy” is presented as a fundamental binary choice that organizations must make as part of their “due diligence” around technology projects.

Here in 2025, this idea is suffering from “concept drift” that makes its simplistic dichotomy a dangerous one, especially for governments.

The Build option

The first problem with “build vs buy” is that “build” doesn’t mean what it used to.

In the past decade or two Open Source Software has “won” and in doing so, fundamentally changed what “building” means. 40 years ago “build” meant tonnes of first-of-it’s kind software painstakingly written in low-level languages, but these days programmers are building with bricks not sand.

Back in 2018 Laurie Voss, the co-founder of the npm package registry captured this shift when he did some analysis showing that 97% of code in modern applications comes from npm. Buy pulling and assembling these packages developers are able to write just the last 3% of the code needed to connect everything together while generating 100% of the value.

The big idea here is that applications are now composed or assembled like a prefab house rather than built from scratch. While prefab components are recognized as a way to mitigate risk in construction, senior executives “limited exposure and experience with information technology projects” (OAG report pg 10) has meant that no such acknowledgement has happened above the working level of the Canadian government.

Without that acknowledgement, while web applications only take a couple of months to build (even for teams in the government!) the “risk management” process around them continues to look like the Manhattan project: budgets in the millions and multi-year, multi-year PowerPoint-driven gated processes full of earnest “go/no-go” decisions, tonnes of scope-creep to “align” with tech decisions past and present, HIPPO effects and a mandatory 8+ month compliance audit.

With the “risk management” process itself adding more cost, time and risk to software projects than the actual software, the “build” option continues to be an option of last resort in spite of an $8.8 trillion ecosystem of open source components readily available.

While the executive class in the Canadian government largely has no idea about this shift from custom code to composition, private industry is well aware of it. This adds a frustrating twist to the “build vs buy” conversation: Executives often avoid “build” because of perceived risk, forbidding employees to assemble the Open Source components themselves in favour of contractors who then assemble the same components for 60% more cost.

This ends up meaning that the “build vs buy” is no longer about “open source vs closed source”, it’s about which organization the skills to competently navigate the Open Source Ecosystem will exist in.

The buy option

As Thoughtworks points out, “a commodity capability will be provided under the assumption that you will adapt your processes to that particular vendor’s definition of industry ‘best practice’”, but the core function of government – governing – is inherently unique to each nation.

With 5,162 municipalities in Canada, this is a big enough customer base that competitive markets can emerge, driving down costs, preventing capture and spreading the “best practices” that software embeds. This is the world the “buy” option is intended to be exercised in.

Where “buy” breaks down

At the Federal level its more complicated. Federal governments are unique in the literal sense; there is only one per country. Each is shaped by a set of laws and policies unique to that countries historical and political context. Software that implements federal “business logic” will have a market of one (a competitive market can’t be created) which suggests that in general, you’d expect to find more custom software at the federal level.

This doesn’t mean “buy” is off the table; Technically savvy governments are able square their unique needs with the imperative to support the free market by breaking down large systems full of custom logic into smaller components, finding generic ones like like sending an email, authenticating users, or managing a wait queue that can easily adapt to a vendors definition of ‘best practice’ and buying those. Commercial cloud providers provide hundreds of commodity services at exactly this level of granularity to allow for this. Again the result is a “composed” system; some amount of custom logic and whatever code is needed to integrate some number of third party services.

It’s unfortunately common in the Canadian federal government that executives don’t think about systems a granular enough level, and often insist on the “buy” option for entire systems, even in the face of unique requirements and processes it either can’t or won’t adapt.

The canonical example is Phoenix, where, despite the complexity (22 different employers in the core public service and agencies, 80 collective agreements and 80,000 business rules) was somehow considered an obvious candidate for “commercial off-the-shelf” software (Phoenix was “Probably destined for failure the minute they made the decision to go with off-the-shelf software” explains André Boudreau, reflecting on lessons from the failed 1995 attempt to replace the pay system).

The government refused to adapt any of it’s practices to fit the “commodity capability” they were buying, but charged ahead anyway, ordering IBM to make 1500 changes which involved rewriting significant parts of the software. This spiraled into a $3.5 billion fiasco that has yet to be fixed (2025 update: Now $5.1 billion!).

Despite the lessons learned from the failed 1995 attempt to replace the pay system saying that the pay rules and processes would need to be simplified before attempting a new system, the choice of a replacement for Phoenix seems to be based on promises that the government won’t need to adapt their practices which unsurprisingly don’t seem to be panning out. With a favorite vendor chosen from a brutal process nobody will be willing to repeat, it appears we are on track to learn this lesson again.

From the looks of it, we have a complex, unique need that requires custom software , but the government has neither the skills to build it, nor the skills to contract it out.

Somewhat unsurprisingly, the skills needed to build and the skills needed to contract it out are related, and over-use of the “buy” option may well be eroding both.

That outsourcing leads to deskilling is seemingly well established, with Dell’s self-inflicted wounds as a cautionary tale from the private sector.

Some years ago, Dell began outsourcing manufacturing to a Taiwanese electronics manufacturer, ASUSTek. The outsourcing started with simple circuit boards, then the motherboard, then the assembly of the computer, then the management of the supply chain and finally the design of the entire computer.

…

The end result? ASUSTeK became Dell’s formidable competitor, while Dell itself, apart from its brand, was hardly more than a shell, without any real expertise to run or grow its business.

The Canadian Government has been aggressively following Dell’s example for years now, and here in 2024 Government executives spent a record amount on outsourcing, with the American government on very similar path.

NASA recently published “NASA at a Crossroads” talking about some of it’s struggles with the deskilling effects of outsourcing. They write of one contract that “In this case, NASA is more of a contract monitor than a technical organization capable of taking humanity into the solar system”, and explain that generally over-use of contracting will “erode the agency’s in-house capabilities”.

Connecting the dots, NASA points out that doing so also effects their contracting ability, and ultimately their mission: “the concern is not only an erosion of “smart-buyer” capability but also of the capacity to invent and innovate”. Anyone who has worked in the Canadian Government will likely recognize that state.

Meanwhile at Canada’s Treasury Board, the 2023 cloud strategy principle #5 is still pushing outsourcing by Prioritizing “buy before build”, thus deskilling their staff (turning them into “contract monitors”), while principle #6 says they want to re-skill staff seemingly without ever wondering why they need to do that.

With the need to form coherent regulations on technical topics like AI, cyberwarfare, crypto-currencies, ransomware, along with a burning need to address the aging IT infrastructure that was identified 24 years ago… there is a deep need in government for technical expertise.

Despite the bad outcomes, somehow the Canadian Government remains bent on following Dell by outsourcing not just technology but policy until it is also just a shell without any of the expertise needed for it’s business of governing. If the handling of Bill C-18 isn’t enough to suggest a problem, remember that the government hired a consultant to figure out how to save money on consultants.

The underlying problem here isn’t buying things, it’s that losing touch with the technical skills needed to build things is setting the stage for both internal facing disasters like Phoenix and outward facing ones, in the form of poor service delivery and ignorant and damaging regulation that makes things worse for business.

Why rehash this now?

The fundamentals of governing depend on basic capabilities like being able pay the public service, or safeguard citizens data. Doing these things increasingly requires things government can’t buy, thus requiring a builders skill set.

Zero Trust is the poster-child for this problem: The US has recognized that the Federal Government can no longer depend on conventional perimeter-based defenses to protect critical systems and data, and ordered the entire US Federal Government to adopt the Zero Trust model. After an uncomfortably long pause Canada has decided to follow (see the Enterprise Cyber Security Strategy and Cloud Adoption Strategy).

Unfortunately, “Zero Trust is a strategy. It’s not a product. You can’t buy it”.

Without a product to buy, it is going to take a builders skill set to consistently interpret and implement Zero Trust principles across all architectural layers in a complex environment full of legacy systems.

After years of outsourcing and deskilling it’s unclear that the Canadian Government is capable of implementing this security model (especially when each pillar is assigned matrix-style to a different group with differing priorities and wildly different levels of technical skills).

Without it, the government’s ability to protect it’s citizens data is suspect, and the willingness for allies to share data with Canada will steadily decrease as Canada fails to implement what they now consider “basic cyber hygiene“, all of which threatens the basic functioning of government.

From build last to something new

The old “build vs buy” debate needs an update. Beyond the basic observation that folks responsible for IT projects should have actual skills in this area, people in government need to wrap their heads around how the two sides of this little dichotomy have shifted if they want better outcomes.

On the “build” side, the time and cost involved are a fraction of what they were, the risk management process doesn’t reflect it. The inability to automate key services causes political crises, and the inability to implement key enabling architectures like Data Mesh and Zero Trust (neither of which can be purchased) setting us up for future crises. Weighing “build” against “buy” without considering how “build” is entwined with these broader issues is damaging.

On the “buy” side, granular services now exist that can allow smart use of commodity components inside larger custom software efforts, but techniques that would help like Wardley mapping are basically unheard-of in the Canadian Government. Using those techniques to safely outsource commodity capabilities also requires a builders skill set.

With service levels directly connected to trust in the government and the CIA noting that “in coming years, this mismatch between governments’ abilities and publics’ expectations is likely to expand and lead to more political volatility“, Treasury Board needs to recognize that the inability to build is a threat to the most basic functions of government.

While there are legitimate moments for the “build vs buy” question, without knowing how things are actually built, having the capability to actually pull it off and the consequences of too much buying, this discussion is just faux diligence that does more harm than good.

TBS’s current “buy before build” mandate makes building an act of last resort, suppressing that critical skill set when it’s needed most.

Instead of “buy before build” we probably need to remember what NASA’s Wernher von Braun had figured out in 1964:

“A good engineer gets stale very fast if he doesn’t keep his hands dirty . . . it is for this reason that we are spending about 10 percent of our money in-house; it enables us to really talk competently about what we are doing. This is the only way known to us to retain professional respect on the part of our contractors.”

We managed to get from “cloud first” to “cloud smart“. Maybe something like “build smart, buy smart” is possible too.

Enterprise architecture (EA) is a troublesome discipline. I think it’s fair to argue that the famous Bezos API mandate and the birth of AWS are both essentially enterprise architecture efforts as is Simian Army from Netflix. These efforts have clearly delivered a huge positive business impact, but it’s much harder to make that case for the version of EA that exists in government.

For this government version, if we look beyond the tendencies towards self-referential documentation, and the use of frameworks that lack empirical grounding, there is an increasingly visible conflict with a growing body of knowledge about risk and resilience that is worth considering.

In Canada, Treasury Boards Policy on Service and Digital requires the GC CIO to define an enterprise architecture while the Directive on Service and Digital requires the departmental CIOs to “align” with it.

EA is used to design a target architecture, but more people are familiar with it as a project gating mechanism where the pressure to “align” is applied to projects. Mostly this takes the form of EA arguing for centralization and deduplication largely justified in terms of imagined cost savings (despite the fact that the combination of virtualization and usage-based billing has largely eliminated the benefits of shared infrastructure).

This focus stands in sharp contrast with the literature on resilience, which largely views this sort of cost-optimization activity as stripping a system of it’s adaptive capacity.

What’s common to all of these approaches- robustness, redundancy, and resilience, especially through diversity and decentralization- is that they are not efficient. Making systems resilient is fundamentally at odds with optimization, because optimizing a system means taking out any slack.

Deb Chachra, How Infrastructure Works

Since this stuff can feel pretty abstract, we can try to make this concrete with a look at Treasury Board’s Sign-in-Canada service which is a key part of their “target enterprise architecture”.

The Government of Canada has ~208 Departments and Agencies, 86 of which have their own accounts. This is often held up as an example of of inefficiency and duplication, and the kind of thing that EA exists to fix. As TBS describes: “Sign‑in Canada is a proposal for a unified authentication mechanism for all government digital engagement with citizens.”

If you skip past the meetings required to get all 86 systems to use Sign-in-Canada, the end result would be a “star graph” style architecture; Sign-in-Canada in the center, with digital services connecting to it.

Prized for efficiency and especially for central control this star-graph style architecture shows up everywhere in governments. To get to this architecture, EA practitioners apply steady pressure in those meetings (those gating functions of Enterprise Architecture Review Boards) to avoid new sign-in systems and ensure new and existing systems connect to/leverage Sign-in-Canada.

In graph theory there is a term for networks that are formed under such conditions; “preferential attachment“, where new “nodes” in the network attach to existing popular nodes.

Networks formed under a preferential attachment model (called “scale-free” in the literature) have some really interesting (and well studied) properties that I think are exactly what EA is trying to encourage; networks formed like this are surprisingly robust to random failures.

If you imagine the power/cooling/rack space constraints of a traditional physical data center, and the challenge of staying within those limits while limiting the effects of random failures, the centralization/deduplication focus of EA is a huge benefit.

But “scale-free” networks also have another property: They are very fragile to targeted attack. Only a handful of highly connected nodes need to be removed before the network is completely destroyed. If targeted attacks are suddenly the concern, the preferential attachment playbook, starts to look like a problem rather than a solution.

It’s these ideas that show why an EA practice narrowly focused on reuse/centralization/deduplication ends up conflicting with resilience engineering and modern security architecture.

Through that resilience lens, the success of Sign-in-Canada means a successful hack (the Okta breach gives us a preview) could paralyze 86 government organizations, something that isn’t currently possible given the redundancy of our current “inefficient” system.

In academic terms what we’ve done is increase our systems “fragility”, it’s a well known byproduct of the kinds of optimizations that EA is tasked with making.

We need to understand that this mechanistic goal of optimization as creating this terrible fragility and that we need to try and think about how we can mitigate against this.

Paul Larcey: Illusion of Control 2023

These system/network properties are well known enough that the US military has developed an algorithm that will induce fragility in human organizations. It uses this to make networks (terror networks in their case) more vulnerable to targeted attack.

The algorithm is called “greedy fragile” and it works by selecting nodes for “removal” via “shaping operations” (you can imagine what removing someone from a social network means in a military context), so that the resulting network is more centralized (“star-like”) and fragile; centralizing as a way maximize the impact of a future attack.

While it might sound uncharitable to lay the responsibility for systemic fragility at the feet of enterprise architecture it is literally the mandate of these groups to identify and make many of these optimizations happen. It’s worth saying the executives fixated on centralization and security’s penchant for highly centralized security “solutions” are big contributors too.

I would argue the 2022 hack of Global Affairs which brought down the entire department for over a month is an example of of this fragility. When an entire department can fail as a single unit, this is an architectural failure as much as it is a security failure; one that says a lot about the level of centralization involved.

It’s worth saying that architecting for resilience definitely still counts as “enterprise architecture”, and in that way I think EA is actually more important than ever. However as pointed out in How infrastructure Works, it would be a big shift from current practice.

“Designing infrastructural systems for resilience rather than optimizing them for and efficiency is an epistemological shift”

Deb Chachra, How Infrastructure Works

We very much need EA teams (and security architecture teams) to make that shift to assuming targeted attacks and focusing on resilience. The EA folks I’ve met are brilliant analysts and more than capable of updating their playbooks with ideas from complex systems, cell-based architecture, resilience patterns like the bulkhead pattern, chaos engineering, or Team-Topologies and using them to build more resilient architectures at every level: both system and organizational.

With Global Affairs, FINTRAC and RCMP all hit within a few weeks of each other here in early 2024, making resilience a priority across the government is crucial and there is nobody better placed to do that than enterprise architects.