Vincristine is a chemotherapy drug that is supposed to be administered into a vein, but it's also usually prescribed alongside other drugs that must be administered into the spine. For decades medical staff would sometimes accidentally inject Vincristine into the spine, which always causes either death or permanent paralysis.
When errors are like this are made people almost always jump to blame the human who made the error, and this situation was no different. Medical staff were disciplined or charged with criminal misconduct, and the only changes made to prevent future errors were new safety guidelines, training, and different labeling. These measures didn't work, and this tragic mistake was made repeatedly for decades.
Eventually someone figured out this entire error could be made impossible if Vincristine was just packaged differently. IV drip bags physically can't be attached to the kind of needles always used for spinal injections, and institutions that made this change have don't make this error anymore.
Don't blame the person, blame the system
These same kinds of systems problems are everywhere. The most important example is the systems we use for voting.
Do you have the intuition that "voting" is when everyone gets to choose just one candidate and the candidate with the most votes wins? I'm here to tell you that isn't the only way to vote, and in fact it's one of the worst ways.
small groups of people can "just figure it out", socializing, reputation, etc, everyone can work together and get on the same page at scale, with large groups of people, we can't do that. we have to have some algorithm, some system of rules that determines how we make decisions, otherwise no collective decision will happen and everyone will just do their own thing.
in order to work together in large groups to complete large projects (companies, countries, cities), we need systems. the design of those systems matters, it changes outcomes.
the rules we use to make decisions (the promises we make to each other about how decisions will be made) matter. they can completely change the actual decision we end up making. the problem isn't that everyone just has the wrong opinions, the problem is the way we aggregate people's opinions
designing a social system is an inherently technical act. a social system has a design, and that design matters
"there's no such thing as human error" when something goes wrong, we're often tempted to blame individuals, bad actors, dumb voters, whatever but the choices people make are always being pushed by the systems they live in. humans will be always fallible, always make mistakes, always susceptible to being tricked or manipulated. if you actually want to solve problems, you have to change systems. https://pmc.ncbi.nlm.nih.gov/articles/PMC1117770/
I call this Systems Realism
The person approach focuses on the errors of individuals, blaming them for forgetfulness, inattention, or moral weakness. The system approach concentrates on the conditions under which individuals work and tries to build defences to avert errors or mitigate their effects.
Blaming individuals is emotionally more satisfying than targeting institutions. People are viewed as free agents capable of choosing between safe and unsafe modes of behaviour. If something goes wrong, it seems obvious that an individual (or group of individuals) must have been responsible.
We cannot change the human condition, but we can change the conditions under which humans work
active failures are like mosquitoes. They can be swatted one by one, but they still keep coming. The best remedies are to create more effective defences and to drain the swamps in which they breed. The swamps, in this case, are the ever present latent conditions.