A/B testing is an effective approach to use science to design and deliver deeply-frustrating user experiences.
A/B testing without upfront research is just random monkeys testing random designs to see which of those designs do “best” against random criteria.
If drug testing was actually implemented like most A/B tests, you’d give 2 drugs to 2 groups of people and pick the “winner” by whichever group had fewer deaths.
The consequences of this design should be obvious. The group’s opinion will converge on that of the highest authority present.
As soon as an authority of any kind makes their opinion known, the group will shift in that direction. Even the most rational will tweak their responses after that. After all, who wants to risk going up against an authority? Interns will hesitate to comment. All objections will be a little bit more qualified or toned down.
Generally speaking, if you are writing a document and want to get the most out of a group’s feedback, each contributor should be able to form their opinions independently and give their responses without fear of social or community repercussions.
Organising information so that it’s easy for a group of people to find the documents they need is very hard.
The alternative is to solve it the same way we did with email: shared data, individual organisation.
You don’t need to know how your colleagues organise their email. You only need to know that they get it and respond. The same applies to most work documents. In Personal Information Management (PIM) this is often called “the user-subjective approach”.