Jethro Lloyd, CEO and President of iLAB.
Large technology modernization programs rarely struggle because of one bad decision or one vendor. More often, based on my experience working with clients on these projects, the warning signs of trouble accumulate over time.
• Scope expands. With large transformations spanning multiple systems, business functions and teams, user acceptance testing often uncovers long-term needs.
• Testing gets compressed to protect the schedule. Pressure to release has a common victim: the manual regression test. Many of our clients, for instance, ship defects where simple automation could greatly improve quality.
• Defects and new requirements become harder to distinguish from each other. After a recent transformation project failure due in large part to uncaptured requirements and unmet needs (reported as defects or bugs), a state agency engaged us to ensure procurement-ready requirements. I recently had the pleasure of presenting to the Florida House Information Technology Budget and Policy Subcommittee 306 detailed business requirements that were validated through 17 workshops with stakeholders and providers.
As go-live approaches, leadership often relies on the same delivery ecosystem to report whether the system is actually ready. These patterns have surfaced repeatedly in public audits, legislative reviews, litigation and post-implementation assessments of large government and enterprise technology programs.
While the names and technologies change, the underlying delivery risks are remarkably familiar.
Scope growth becomes difficult to separate from rework.
Change is inevitable in a multi-year modernization. The problem begins when leadership loses visibility into why the scope is changing.
Is a contract amendment funding a genuinely new business requirement? An unforeseen dependency? A requirement that was missed during discovery? Or remediation of something that should have been delivered previously?
Those distinctions become increasingly difficult to make without insights into:
• Requirements-To-Test Traceability: Which requirements have no associated test coverage?
• Change-Impact Validation: If this requirement changes, what else could break?
• Clear Separation Of Defects, Enhancements And New Scope: Is this actually a defect? Or is the system behaving as designed?
• Quality Metrics That Show Where Risk Is Accumulating: Which areas have the least test coverage relative to their business criticality?
The program discovers complexity too late.
Legacy modernization almost always uncovers complexity that was not completely visible at procurement. Interfaces, data dependencies, business rules and operational processes begin colliding as the program moves closer to production.
A recent statewide public sector Salesforce implementation, for example, had to account for regulations, payments, integrations and legacy data migration. We combated delays with validating data migration, third-party integrations, business functionality, UAT and regression testing, paired with phased releases.
Testing is often the first time assumptions finally meet reality. Testing business-critical workflows, integrations, data and end-to-end processes earlier gives leadership an opportunity to address gaps while there is still time to act.
For example, testing the whole quote-to-cash workflow might uncover an approval rule, pricing calculation or handoff that doesn’t work before it impacts the bottom line. Or, by testing a process such as hire-to-retire across SuccessFactors, payroll, benefits, identity/access and downstream reporting, you can spot integration gaps as the business process crosses application boundaries.
Testing becomes a scheduled activity instead of a readiness measure.
Leadership needs to know what was actually tested, which business processes remain exposed, what defects are still open, whether regression coverage is sufficient and whether agreed exit criteria have genuinely been met.
That requires quality metrics tied to readiness, rather than simply reporting how many test cases were executed: for example, the percentage of requirements with test coverage, the percentage with traceability to business objectives and the percentage with approved acceptance criteria. Together, these help answer whether what’s being built is defined well enough to validate.
Delivery assurance can be approached holistically.
Gaining visibility into modernization risk requires more than adding another round of testing. Organizations need a delivery assurance strategy that connects requirements, risk, quality, testing, data, integrations, business readiness and executive decision making.
Consultants can play an important role in that strategy. My firm, iLAB, is one example of this kind of independent partner that sits alongside the system integrator and internal delivery teams to give leadership an unaffiliated source of evidence on program readiness.
The value of independence is not simply having another team test the system, but in creating a perspective that is not responsible for delivering the solution and, therefore, can challenge assumptions about whether the evidence actually supports a release decision.
Organizations can establish this same visibility in several ways:
• Internal QA Or Quality Center Of Excellence: Organizations with mature QA capabilities can establish their own independent governance, standards, traceability, metrics and readiness reviews.
• Independent Verification And Validation (IV&V): Common in highly regulated environments, IV&V provides an independent assessment of whether the program is meeting requirements and managing risk.
• Independent Delivery Assurance: A broader model that evaluates requirements, program governance, risk, testing, data, integrations, operational readiness and go-live evidence, but operates separately from the project delivery team.
• System-Integrator-Led QA: The implementation partner can own quality and testing, particularly when the organization has strong internal governance and sufficient independent oversight.
Each approach has trade-offs. Embedding quality entirely within the implementation team can provide strong domain knowledge and efficiency, but may limit independence. Building an internal QA organization provides institutional knowledge and long-term capability, but requires people, processes and expertise to challenge objectively. An external partner provides independence and specialized expertise, but needs clearly defined responsibilities and access to the information required to be effective.
The common pitfall is treating independence as the objective rather than visibility and evidence as the objective. The program should have enough visibility to recognize those signals and enough delivery discipline to act on them. The goal is not to find fault with the implementation partner, but to find risk before the business does.
Large modernization programs need strong system integrators. They also benefit from an independent quality perspective that asks a different question. When critical services, major investments and public trust are on the line, answering, “Do we have the evidence to be confident this system is ready?” is just as important as answering, “Are we on schedule?”
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?