Notes on the Legibility War An Article by David R. MacIver notebook.drmaciver.com The basic idea of legibility is that the act of making something comprehensible enough to control is itself an act that shapes the thing to be controlled, often with far greater consequences than the control itself. This is because it removes complexity that is deemed as irrelevant that makes it harder to control, and that complexity may be in some way essential to the health of the system. controlsystemscomplexitylegibility
Lost purposes There’s chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys. If you find there’s no chocolate at the supermarket, you won’t stand around opening and slamming your car door because the car door still needs opening. I rarely notice people losing track of plans they devised themselves. It’s another matter when incentives must flow through large organizations—or worse, many different organizations and interest groups, some of them governmental. Then you see behaviors that would mark literal insanity, if they were born from a single mind. Someone gets paid every time they open a car door, because that’s what’s measurable; and this person doesn’t care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving. Eliezer Yudkowsky, Rationality: From AI to Zombies So many tactics, so well entrenched metricsgoals