With the resignation of the CIO of the Government of Canada, the person placed at the top of the Canadian public service to fix the existing approach to IT, there is lots of discussion about what’s broken and how to fix it.
Across these discussions, one thing stands out to me: IT security always seems to get a pass in discussions of fixing/modernising IT.
This post is an attempt to fix that.
As the article about the CIO points out, “All policies and programs today depend on technology”. IT Security’s Security Assessment and Authorization (SA&A) process applies to all IT systems therefore landing on the critical path of “all policies and programs”. This one process adds a 6-24 month delay to every initiative and somehow escapes any notice or criticism at all.
If you imagine some policy research, maybe a public consultation and then implementation work, plus 6-24 months caught in the SA&A process, it should be clear that a single term in office may not be enough be able to craft and launch certain initiatives let alone see benefits from them while in office. Hopefully all political parties can agree fixing this is in their best interests.
As a pure audit process, the SA&A is divorced from the technical work of securing systems (strangely done by operations groups or developers, rather than by security groups) leaving lots of room to reshape (or eliminate) this process without threatening actual security work. Improvements in this process are probably the single most impactful change that can be made in government.
It’s also key to accelerating all other modernisation initiatives.
Everyone in Ottawa is well aware that within each department lies one or more likely political-career-ending ticking legacy IT timebombs. Whether this is the the failure of the system itself, or of the initiative launched to fix it, or even just the political fallout from fixed capacity systems failing to handle a surge in demand, every department has these and the only question is who will be in office when it happens.
Though you’d never guess, inside the government it is actually known how to build systems that can be modernised incrementally, changed quickly, rarely have user visible downtime and can expand to handle the waves of traffic without falling over.
The architecture that allows this (known as microservices) was made mandatory by TBS in the 2017 Directive on Service and Digital. The Directives successor (the Enterprise Architecture Framework) doesn’t use the term directly but requires that developers “design systems as highly modular and loosely coupled services” and several other hallmarks of microservices architecture that allow for building resilient digital services.
I think TBS was correct in it’s assessment that this architecture is key to many modernisation initiatives and avoiding legacy system replacements just as inflexible as their predecessors. Treasury Board themselves describes the difference between current practice as their target architecture as a “major shift” but the number of departments willing/able to make that shift hovers close to zero.
AWS uses the same microservices architecture to deliver their digital services and promotes it, along with the infrastructure and team structures needed to support it, under the banner “Modern Applications“. Substantially similar advice is given by Google and others and these best practices have been worked into TBS’s policy since 2017.
While much of TBS IT policy has been refreshed, all core IT security guidance and patterns are built around pre-cloud ideas (ITSG 22 from 2007, ITSG 38 from 2009) and process (ITSG 33 from 2012).
While TBS might want departments to adopt microservices (created circa 2010 around the “death” of SOA), it’s the 1990s-era 3-tier architecture (what ITSG-38 still calls the “preferred architecture”) that the network and security infrastructure is set up to support, rather than the fancy compute clusters and cloud functions needed for the microservices architecture; an application architecture that exists to fix the very visible problems governments (and others) have with availability and scalability.
Unfortunately, these legacy design patterns mean that security teams spend months or years building out a digital copy of a legacy Data centre in the cloud. These come with complex routing and subnets with a fixed-capacity Virtual Machine based security appliance at the center.
Beyond the staggering delay, the result is an anti-pattern where the network layer cancels out the scalability and availability benefits of the application architecture above and the cloud architecture below.
These fixed-capacity virtual appliances can’t handle waves of traffic or scale to zero like AWS Lambda or Google Cloud Run, and the work that TBS is hoping will go into creating those high-availability microservices just won’t matter when security brings down the firewall in front to patch it.
Similar conflicts exist with TBS’s requirements for the Dev(Sec)Ops and “multidisciplinary teams” practices needed for this architecture. These practices are likely not feasible given almost every action currently requires manual security approvals, or the current interpretations and enforcement of ITSG-33’s AC-5 “separation of duties” and the naive insistence that distant monitoring/SoC teams operating with zero context can somehow parse meaning out of the complex communication patterns of microservices applications.
While TBS has updated it’s policy to require agile development practices, ITSG-33, the foundation of all government security process is explicitly waterfall, and while adapting it to agile is theoretically possible, it’s developers that get exposure to agile methods, rather than the well intentioned auditors and former network admins that populate most security groups.
Beyond the shift to agile, the division of labour (the split between dev/sec/ops) systemically undermines ITSG-33’s approach: the technical work of securing systems builds security engineering expertise in operations groups, while the spreadsheet-based audit work of security teams chases away technical talent and steadily erodes any remaining technical capacity.
This dynamic reliably creates a skills imbalance that undermines the audit/watchdog role ITSG-33 imagines for security teams; The 2019 State of DevOps report noted a similar effect with CoEs: “This disconnect between theory and hands-on practice will eventually threaten their expertise”. Luck and smart hiring can help, but security teams are forever swimming against this current. When you mix security and compliance, what survives is compliance.
The failure case is the familiar security-theater of the audit process: non-technical auditors “verifying” the work of developers and operations via screenshots without being able to read or run code and executives placing their faith in this paperwork rather than the judgement of the dev/ops teams doing the engineering work of securing systems.
Typically contractor driven, each one costs over $70,000 and at least 6 months for a paper audit of a simplistic one-piece monolithic application. Nobody is sure how to apply this process to the microservices architecture TBS is pushing where each application is made of dozens, or even thousands of separate applications.
This process is almost single-handedly responsible for the lack of progress on modernisation: Since all departments have hundreds of applications, assuming an optimistic audit timeline of 6 months each, even the most milquetoast modernization effort implies a SA&A/audit process bottleneck measured in decades.
Surrounded by hollowed-out waterfall security processes and pre-cloud security architecture that no-one seems equipped to change, Treasury Board’s vision of the government delivering modern, reliable digital services flounders.
The idea here is that in many cases modernising security is a precondition to successfully modernising anything else. For those that overlook security, the assumptions embedded in their tools, processes and architectures will subtly but steadily undermine their efforts. Treasury Board is filled with smart policy analysts learning this the hard way.
Security is the base of the modernisation pyramid… start there to fix things.