MCX Services โ€” Thought Leadership

Software's Safety Culture Problem

Aviation built its safety culture after Tenerife. Nuclear built its after Three Mile Island. Medicine built its after countless preventable deaths. Software is still waiting for its reckoning. Vibe coding may be accelerating the timeline.

๐Ÿ“– 13 min read โœˆ๏ธ Cross-industry analysis ๐ŸŽฏ For CTOs, Engineering Leaders & the Industry at Large

Every industry that takes safety seriously got there the same way. Not through gradual, proactive improvement. Through catastrophe. Through a failure so visible, so costly, or so lethal that the old way of operating became politically and culturally impossible. The new safety culture did not emerge from enlightenment. It emerged from wreckage.

Aviation had Tenerife in 1977, where two 747s collided on a runway and killed 583 people, triggering a complete redesign of cockpit communication norms, crew resource management training, and authority gradients. Nuclear had Three Mile Island in 1979 and Chernobyl in 1986, triggering independent safety reviews, redundant safety systems, and an entirely new regulatory posture. Medicine had the Institute of Medicine's "To Err is Human" report in 1999, which documented 98,000 preventable deaths annually in U.S. hospitals and triggered a decade of systemic reform.

Each industry resisted change before its reckoning. Each had advocates who argued that existing practices were sufficient, that costs of reform were prohibitive, that catastrophic failure was unlikely. Each was wrong. And each, after the reckoning, built safety infrastructure that made the pre-reckoning era look like a different civilization.

Software is still in the pre-reckoning era. The question is not whether a reckoning is coming. It is when โ€” and whether the industry will act before it arrives or after.

What a Safety Culture Actually Requires

A safety culture is not a set of policies. It is a set of structural commitments that change the default behavior of everyone in the system โ€” from the engineer writing code to the executive making release decisions. The industries that have built it share a common architecture.

How Other Industries Built Safety Culture โ€” and What Forced Them To

The pattern is consistent: catastrophe, then structural reform
Aviation
The catalyst
Tenerife (1977): 583 deaths. United 173 (1978): fuel exhaustion because crew deferred to captain. Air Florida 90 (1982): co-pilot failed to challenge captain's decision to take off with iced wings.
The structural response
Crew Resource Management became mandatory. Authority gradients were redesigned. Any crew member gained the explicit right โ€” and responsibility โ€” to challenge any decision regardless of rank. Checklists became non-negotiable.
Nuclear
The catalyst
Three Mile Island (1979): operator actions worsened the accident. Chernobyl (1986): safety test conducted improperly because engineers bypassed safety systems to meet a schedule deadline.
The structural response
Independent Safety Culture Assessment became a regulatory requirement. Operators gained the explicit authority to shut down a plant without management approval. Safety could not be overridden by schedule pressure. The word "never" appeared in procedure manuals.
Medicine
The catalyst
"To Err is Human" (1999): 98,000 preventable deaths annually in U.S. hospitals. Wrong-site surgeries. Medication errors. System failures mistakenly attributed to individual error.
The structural response
Surgical checklists (WHO, 2008) reduced major complications by 36%. Medication barcoding eliminated class of errors entirely. Root cause analysis replaced individual blame. Reporting systems made errors visible and actionable.
Construction
The catalyst
Decades of preventable workplace fatalities. The US construction industry had fatality rates in the 1970s that would be unacceptable in any modern context.
The structural response
OSHA standards, safety officer roles with authority to stop work, mandatory incident reporting, and a cultural shift from "accidents happen" to "accidents are preventable." Fatality rates dropped by more than 60% over four decades.

The pattern in each case is identical. Before the reckoning: safety is someone's job among many, schedule pressure routinely overrides safety concerns, individuals who raise safety issues face professional consequences, and the system is optimized for throughput rather than reliability. After the reckoning: safety becomes a structural property of the system, not a personal responsibility of individuals within it. The change is not cultural in the soft sense. It is architectural.

Where Software Stands

Software has had incidents. It has had breaches, outages, and failures with real human consequences. Patients have been harmed by software failures in medical devices. Financial systems have failed and taken customer assets with them. Critical infrastructure has been compromised. Aircraft flight management software has contributed to fatal crashes.

What software has not had is a single, visible, unambiguous failure large enough and attributable enough to force structural reform across the industry. The failures that have occurred have been absorbed โ€” financially, reputationally, legally โ€” without producing the systemic response that aviation's or medicine's reckoning produced. The industry has been fortunate in this. The fortune may not continue.

$4.2M
Average cost of a data breach in 2024 โ€” and rising. The financial consequence exists. The structural response does not.
IBM Cost of a Data Breach Report, 2024. Does not include regulatory penalties, litigation, or reputational cost.

Why Vibe Coding Accelerates the Timeline

The emergence of AI-assisted development โ€” what the industry has taken to calling vibe coding โ€” changes the risk calculus in ways the industry has not yet absorbed. Not because AI is inherently unsafe, but because it dramatically increases the volume of code entering production pipelines while the verification infrastructure underneath those pipelines has not changed.

The Four Ways AI-Assisted Development Amplifies Software Safety Risk

Volume without proportional verification
AI-assisted developers produce significantly more code per unit time than unassisted developers. The QA infrastructure โ€” test coverage, security scanning, compliance review โ€” has not scaled proportionally. More code with the same verification capacity means lower effective coverage per line shipped. The risk surface expands. The detection capability does not.
Lost author understanding
When a developer writes code by hand, they carry implicit understanding of the design decisions embedded in it. When an LLM generates code that a developer accepts and ships, that understanding does not transfer. The developer cannot answer, under audit or under incident, why the function is structured the way it is โ€” because they did not design it. This is not a criticism of the practice. It is a statement about the verification requirement it creates.
Confident ignorance at scale
AI-generated code is often syntactically correct and functionally plausible. It passes code review on appearance. It passes unit tests that it may have been generated alongside. The failure modes are not visible to casual inspection. They live in edge cases, in timing behavior, in security boundary handling that requires adversarial testing to surface. The code looks right. The confidence is not earned.
Speed pressure on verification
The velocity gain from AI-assisted development creates schedule pressure to ship at the same accelerated pace. Teams that could previously ship a feature in two weeks now ship it in three days. The expectation follows the velocity. The QA cycle โ€” which was already compressed under normal velocity โ€” gets compressed further. The ratio of development speed to verification thoroughness moves in the wrong direction.

"The aviation industry's pre-Tenerife culture had a phrase: 'captain's authority.' The captain's judgment was not to be questioned by crew. Multiple crashes were attributable, in retrospect, to crew members who saw something wrong and did not say it โ€” because the culture made deference to authority the safe personal choice. Software has its own version of this: 'ship it.' The culture makes deference to schedule the safe personal choice. The mechanism is different. The outcome is the same."

โ€” Human factors researcher, aviation safety background

What Software Safety Culture Would Actually Require

The industries that built safety cultures did not do it by asking individuals to be more careful. They built infrastructure that made unsafe behavior structurally difficult โ€” that changed the system so that the safest path and the easiest path were the same path. Software safety culture, when it arrives, will require the same approach.

The Structural Properties of a Software Safety Culture

Verification as default, not exception
Every commit is tested automatically. Every dependency is scanned. Every compliance surface is checked. Verification happens by default at the point of creation โ€” not as a phase at the end of a sprint or a gate before a release.
Objective release criteria
Release decisions are made against quantified thresholds โ€” coverage percentage, open CVE count, compliance control status โ€” not against team confidence. "Do we feel ready?" is replaced by "do we meet the criteria?"
Protected dissent
Any engineer can flag a quality concern without career risk. The QA lead who says "coverage is at 47% on the payment module" should not need political courage to say it. The data says it. The person presents it. The decision belongs to leadership.
Continuous evidence trail
Every release decision is documented with the quality evidence available at the time. Post-incident analysis starts from "here is what was known" rather than "here is what we assumed." Accountability is grounded in fact, not reconstruction.
Schedule subordinated to safety thresholds
The release criteria exist regardless of what marketing scheduled. A feature that does not meet coverage thresholds does not ship โ€” the same way a nuclear plant that does not pass safety inspection does not restart, regardless of the energy demand that day.
System accountability, not individual blame
When a defect escapes, the post-mortem asks what the system allowed rather than who made the error. The answer almost always reveals a gap in the verification infrastructure โ€” a coverage hole, a missing scan, a compliance surface that was never checked. Fix the system. Not the person.

None of these properties require a regulatory mandate. They require a decision by engineering leadership to build the infrastructure before the reckoning rather than after it. Aviation did not wait for regulators to invent Crew Resource Management โ€” the industry built it because the alternative was continued catastrophe. The software industry can make the same choice.

Post-incident reform
Pre-incident design
Safety Culture Timing
Individual responsibility
System design
Safety Accountability Model
Confidence-based release
Evidence-based release
Release Decision Foundation
Waiting for reckoning
Acting before it
Industry Position

The Bottom Line

The software industry is not uniquely reckless. It is where every high-velocity, high-consequence industry was before its safety culture moment โ€” optimizing for throughput, absorbing failures as individual events rather than systemic signals, and deferring the structural reform that would prevent future failures until those future failures become impossible to absorb.

The difference is that software's failures are increasingly consequential. The systems it builds are increasingly critical. The pace of AI-assisted development is increasing the volume and velocity of code entering production. And the verification infrastructure underneath all of it has not changed in proportion to any of these trends.

The reckoning may be a single visible catastrophe. It may be the cumulative weight of a thousand smaller failures. It may be regulatory action that arrives faster than the industry expects. Whatever form it takes, the organizations that will navigate it best are not the ones that responded to it โ€” they are the ones that built the infrastructure before it arrived.

Aviation did not need Tenerife to tell it that cockpit communication was important. It needed Tenerife to act on what it already knew. Software already knows. The question is whether it will act.

Build the Infrastructure Before the Reckoning.

MCX Services helps engineering organizations build the quality infrastructure that a genuine software safety culture requires โ€” continuous verification, objective release criteria, and evidence-based decision making. The conversation starts with where your organization stands today.

Software's Safety Culture Problem: What Aviation and Medicine Learned That Software Has Not | MCX Services | MCX Services