High levels of U.S. patient satisfaction are mainly associated with hospitality (greeters at the door, empathetic staff, comfortable rooms) – but also with more treatments, high costs, and substantially higher mortality even after adjusting for baseline health and comorbidities. Several plausible stories explain these big n and replicated observational findings. Whatever the case, post-treatment patient satisfaction/gratitude does not measure whether a treatment works or not. Patient outcomes decide.
The Silicon Valley giants, testifying with their runaway success, claimed to have “solved” design as an engineering problem. The solution substituted the human essence of design — intuition, ingenuity, and taste— with the tangibles, measurables, and deliverables.
Companies say they are “design-driven”, but designers are actually driven by dashboards filled with metrics like CSAT, NPS, CES, DAU, MAU. We rigorously run tests, studies, experiments as if innovative ideas are hidden in spreadsheets, waiting to be extracted by data scientists.
Modernist planning was obsessed with absolute numbers, including the minimum dimensions of rooms, open space per capita, and the one-size-fits-all head counts of neighborhood units. This was often pegged at five to seven thousand and was used as a formula for determining the distribution of schools, shops, sports fields, and other facilities. The failure of such planning is not in its effort to be comprehensive or to equalize access to necessary facilities. It is, rather, the attempt to rationalize choice on the basis of a homogeneous set of subjects, a fixed grammar of opportunities, a remorseless segregation of uses, and a scientistic faith in technical analysis and organization that simply excludes diversity, eccentricity, nonconforming beauty, and choice. The utopian nightmare.
There’s chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys. If you find there’s no chocolate at the supermarket, you won’t stand around opening and slamming your car door because the car door still needs opening. I rarely notice people losing track of plans they devised themselves.
It’s another matter when incentives must flow through large organizations—or worse, many different organizations and interest groups, some of them governmental. Then you see behaviors that would mark literal insanity, if they were born from a single mind. Someone gets paid every time they open a car door, because that’s what’s measurable; and this person doesn’t care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving.
Analytics apps don’t tell you much about usage behavior. You might be able to see how many users performed an event, or how many times they did it. But none of the analytics packages out there are good at showing you how often people do things. Are they using to-dos once a week? Every day? Only signing into the app once a month but happily paying for years?
Time matters. You can’t understand usage without time.
As we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.
The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.
Once in place, metrics have a strange hold on the imagination: I’ve seriously had a CTO carelessly reject my genuine idea out of hand because "it doesn’t help OKRs", the same OKRs we previously agreed should not describe all that we do.
I agree with Amir Shevat that we should “do the right things over the easy to measure things.”
The product/market fit definitions I had found were vivid and compelling, but they were lagging indicators — by the time investment bankers are staking out your house, you already have product/market fit. Instead, Ellis had found a leading indicator: just ask users “how would you feel if you could no longer use the product?” and measure the percent who answer “very disappointed.”
Site performance is potentially the most important metric. The better the performance, the better chance that users stay on a page, read content, make purchases, or just about whatever they need to do. A 2017 study by Akamai says as much when it found that even a 100ms delay in page load can decrease conversions by 7% and lose 1% of their sales for every 100ms it takes for their site to load which, at the time of the study, was equivalent to $1.6 billion if the site slowed down by just one second.
The McNamara fallacy, named for Robert McNamara, the US Secretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations (or metrics) and ignoring all others. The reason given is often that these other observations cannot be proven.
The fallacy refers to McNamara’s belief as to what led the United States to defeat in the Vietnam War—specifically, his quantification of success in the war (e.g., in terms of enemy body count), ignoring other variables.
The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.
Metrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.
The predicted mean vote (PMV) was developed by Povl Ole Fanger at Kansas State University and the Technical University of Denmark as an empirical fit to the human sensation of thermal comfort. It was later adopted as an ISO standard. It predicts the average vote of a large group of people on the a seven-point thermal sensation scale where:
What are the effects of this enumeration, of these metrics that count our social interactions? In other words, how are the designs of Facebook leading us to act, and to interact in certain ways and not in others? For example, would we add as many friends if we weren’t constantly confronted with how many we have? Would we “like” as many ads if we weren’t told how many others liked them before us? Would we comment on others’ statuses as often if we weren’t told how many friends responded to each comment?
In this paper, I question the effects of metrics from three angles. First I examine how our need for personal worth, within the confines of capitalism, transforms into an insatiable “desire for more.” Second, with this desire in mind, I analyze the metric components of Facebook’s interface using a software studies methodology, exploring how these numbers function and how they act upon the site’s users. Finally, I discuss my software, born from my research-based artistic practice, called Facebook Demetricator (2012-present). Facebook Demetricator removes all metrics from the Facebook interface, inviting the site’s users to try the system without the numbers and to see how that removal changes their experience. With this free web browser extension, I aim to disrupt the prescribed sociality produced through metrics, enabling a social media culture less dependent on quantification.