Our Quantified Life
The primary measure of progress
SAFe is oriented around volume, not value
Outcomes decide
Design as an engineering problem
Obsessed with absolute numbers
Lost purposes
The case against heatmaps
It's All Over
An Essay by Justin E. H. SmithIt has come to seem to me recently that this present moment must be to language something like what the Industrial Revolution was to textiles. A writer who works on the old system of production can spend days crafting a sentence, putting what feels like a worthy idea into language, only to find, once finished, that the internet has already produced countless sentences that are more or less just like it, even if these lack the same artisanal origin story that we imagine gives writing its soul. There is, it seems to me, no more place for writers and thinkers in our future than, since the nineteenth century, there has been for weavers.
Time-based analytics
An Article by Ryan SingerAnalytics apps don't tell you much about usage behavior. You might be able to see how many users performed an event, or how many times they did it. But none of the analytics packages out there are good at showing you how often people do things. Are they using to-dos once a week? Every day? Only signing into the app once a month but happily paying for years?
Time matters. You can't understand usage without time.
Fast Path to a Great UX – Increased Exposure Hours
An Article by Jared SpoolAs we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.
The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.
Metrics have a strange hold on the imagination
A Fragment by Shawn WangOnce in place, metrics have a strange hold on the imagination: I've seriously had a CTO carelessly reject my genuine idea out of hand because "it doesn't help OKRs", the same OKRs we previously agreed should not describe all that we do.
I agree with Amir Shevat that we should "do the right things over the easy to measure things."
How would you feel if you could no longer use the product?
An ArticleThe product/market fit definitions I had found were vivid and compelling, but they were lagging indicators — by the time investment bankers are staking out your house, you already have product/market fit. Instead, Ellis had found a leading indicator: just ask users “how would you feel if you could no longer use the product?” and measure the percent who answer “very disappointed.”
Site performance is potentially the most important metric
A Fragment by Kealan ParrSite performance is potentially the most important metric. The better the performance, the better chance that users stay on a page, read content, make purchases, or just about whatever they need to do. A 2017 study by Akamai says as much when it found that even a 100ms delay in page load can decrease conversions by 7% and lose 1% of their sales for every 100ms it takes for their site to load which, at the time of the study, was equivalent to $1.6 billion if the site slowed down by just one second.
The McNamara fallacy
A DefinitionThe McNamara fallacy, named for Robert McNamara, the US Secretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations (or metrics) and ignoring all others. The reason given is often that these other observations cannot be proven.
The fallacy refers to McNamara's belief as to what led the United States to defeat in the Vietnam War—specifically, his quantification of success in the war (e.g., in terms of enemy body count), ignoring other variables.
Artifice, blindness, and suicide
A QuoteThe first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.
Weighing up UX
An Article by Jeremy KeithMetrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.
Predicted Mean Vote
A DefinitionThe predicted mean vote (PMV) was developed by Povl Ole Fanger at Kansas State University and the Technical University of Denmark as an empirical fit to the human sensation of thermal comfort. It was later adopted as an ISO standard. It predicts the average vote of a large group of people on the a seven-point thermal sensation scale where:
- +3 = hot
- +2 = warm
- +1 = slightly warm
- 0 = neutral
- -1 = slightly cool
- -2 = cool
- -3 = cold
What Do Metrics Want? How Quantification Prescribes Social Interaction on Facebook
A Research Paper by Benjamin GrosserWhat are the effects of this enumeration, of these metrics that count our social interactions? In other words, how are the designs of Facebook leading us to act, and to interact in certain ways and not in others? For example, would we add as many friends if we weren’t constantly confronted with how many we have? Would we “like” as many ads if we weren’t told how many others liked them before us? Would we comment on others’ statuses as often if we weren’t told how many friends responded to each comment?
In this paper, I question the effects of metrics from three angles. First I examine how our need for personal worth, within the confines of capitalism, transforms into an insatiable “desire for more.” Second, with this desire in mind, I analyze the metric components of Facebook’s interface using a software studies methodology, exploring how these numbers function and how they act upon the site’s users. Finally, I discuss my software, born from my research-based artistic practice, called Facebook Demetricator (2012-present). Facebook Demetricator removes all metrics from the Facebook interface, inviting the site’s users to try the system without the numbers and to see how that removal changes their experience. With this free web browser extension, I aim to disrupt the prescribed sociality produced through metrics, enabling a social media culture less dependent on quantification.
Seeing With Fresh Eyes
Meaning
Space
Data
Truth
The problem with trees
Many systems are organized hierarchically. The CERNDOC documentation system is an example, as is the Unix file system, and the VMS/HELP system. A tree has the practical advantage of giving every node a unique name. However, it does not allow the system to model the real world. For example, in a hierarchical HELP system such as VMS/HELP, one often gets to a lead on a tree such as:
HELP COMPILER SOURCE_FORMAT PRAGMAS DEFAULTS
only to find a reference to another leaf: Please see
HELP COMPILER COMMAND OPTIONS DEFAULTS PRAGMAS
and it is necessary to leave the system and re-enter it. What was needed was a link from one node to another, because in this case the information was not naturally organized into a tree.
Content-responsive space
Content-responsive spaces in text can be as meaningful as spaces and line breaks in computer code, poetry, math, dialogues, cartoons.
For 1500 years, printed text has used grids indifferent/hostile to meaning. Content-responsive grids are better than imperious grid-possessed layouts. To clarify and intensity meaning, authors and editors can remodel relations between spaces and words...insisting on control of line breaks by authors (who, after all, know the content).
Personal annotations
Yehudi Menuhin, a great violinist, marked up a score for Bach's Sonata No. 2 for Solo Violin. Penciled annotations show real-time performance strategies. To outsiders, insider markups appear chaotic and cryptic, but these personal annotations are for Menuhin's eyes, the only eyes that matter. All can learn from this useful workaday grid strategy: a relevant and intense data layer can become a coherent substrate scaffold upon which to overlay additional information. Maps do this all day long.
What excellence is
Learn what excellence is, how to identify it...This is not a big reading assignment – excellence is scarce, lognormal, long-tailed. Acting on this knowledge is liberating, freeing oneself from vast piles of triviality, knock-offs, petty connoisseurship, over-publishing, and the short-sighted, trendy, greedy. Excellence is long-term knowledge, even forever knowledge.
Excellence, like good taste, is perhaps a universal quality. Analytical thinking is about the relationship between evidence and conclusions, and is fundamental to all empirical work, regardless of field, discipline, specialty. Thus it is possible at times to assess credibility of nonfiction work without being a content expert. Thinking eyes may well have an eye for excellence, regardless of field or discipline.
Sentences and words do not exist by themselves
Sentences and words do not exist by themselves, but have natural, inevitable, unavoidable interactions with their surrounding spaces, words, and other sentences. Sentences are not independent of their spatial context, and interactions can create meanings and harms. Sentences survive content-indifferent and content-hostile spacings, but surviving is not thriving. Text space should not be owned and governed by generic productions grids, which make for convenient production but inconvenient meaning. Space can and should be content-responsive, actively contributing to meaning – forever practices in poetry, maps, math, computer code, comics, theater/movie scrips, posters. Subtle visual spacing differentiates and clarifies sentences, and meaning becomes more consequential, memorable, retrievable.
Central-axis text
Central-axis provides a clear signal of the next line, so that readers and speakers don't have to search on the left margin, sometimes accidentally skipping down a line. Ragged-left typography is used for dialogue in novels and scripts. In central-axis, each line is activated at both left and right margins – unlike squared-off conventional text. Readers/speakers are aware of the length of the next line at both its beginning and end. That knowledge may also help readers detect the pace and rhythm of the words, as in reading poetry aloud.
Idiosyncratic paragraphs
Text-only paragraphs differ from one another only in their words. All the words are typographically the same – typeface, spacings, line-lengths piled up into long deep columns. Systematic regularity of text paragraphs is universally inconvenient for readers, who are unable to find and read once against a specific string of words in previously-read paragraphs. All readers have encountered this problem in essays, articles, novels, news reports. Idiosyncratic paragraphs assist memory and retrieval by readers, by uniquely activating the relevant neural substrates for retaining visual memories. Nearly every paragraph in this book is deliberately unique.
No more LittleDataGraphics
Small data sets should be shown directly...LittleDataGraphics (pie charts, bar charts) translate and encode data into areas and colors. Viewers must then mentally translate codes back into numbers. These codes are unique to the local sets of data graphics, and do not repay learning. Instead, just directly show numbers as numbers. No more LittleDataGraphics. Data visualizations are at their best when there is so much data that the only way to see it...is to see it.
An immense wordy diagram
In ~1560 Ettore Ausonia, a polymath with interests from mathematics to mirror-making, constructed an immense wordy diagram depicting reflections from concave spherical mirrors. Then, between 1592 and 1601, while teaching at the University of Padua, Galileo made this handwritten copy of the diagram, which was fortunate since Ausonio's original has since gone missing. Three helpful architectures for the off-the-grid sentences are deployed – word trees, stacklists, annotated linking lines.
Stacklists
Stacklists organize and clarify complex material in 2-space. Readers read more slowly, and that's good: to think, look again, and connect words vertically within each stack and horizontally between stacks. Instead of polyphony, conventional inline lists are a freight train of words along a one-way narrow track, making it difficult to identify which words belong to which list and to link and compare elements within and between lists.
Observe data collection at the moment of measurement
See, observe, learn how data are collected at moment and place of measurement. "You never learn more about a process than when you directly observe how data are actually measured," said Cuthbert Daniel, a superb applied statistician. See with fresh eyes. Walk around what you want to learn about. Talk to those who do measurements. See how numbers came to be.
Do those measuring know the desired answer? Are those measuring skilled, alert, honest, biased, incompetent, sloppy, tired and emotional?...Artifacts and errors in measurements measured? How are outliers adjudicated?
46 data quality issues in spreadsheets
Outcomes decide
High levels of U.S. patient satisfaction are mainly associated with hospitality (greeters at the door, empathetic staff, comfortable rooms) – but also with more treatments, high costs, and substantially higher mortality even after adjusting for baseline health and comorbidities. Several plausible stories explain these big n and replicated observational findings. Whatever the case, post-treatment patient satisfaction/gratitude does not measure whether a treatment works or not. Patient outcomes decide.
Which half?
One day when I was a junior medical student, a very important Boston surgeon visited the school and delivered a great treatise on a large number of patients who had undergone successful operations for vascular reconstruction. At the end of the lecture, a young student at the back of the room timidly asked, "Do you have any controls?" Well the great surgeon drew himself up to his full height, hit the desk, and said, "Do you mean did I not operate on half of the patients?" The hall grew very quiet then. The voice at the back of the room hesitantly replied, "Yes, that's what I had in mind." Then the visitor's fist really came down as he thundered, "Of course not. That would have doomed half of them to their death!" God, it was quiet then, and one could scarcely hear the small voice ask, "Which half?"
Good annotation
Information displays should be annotated, combining words, images, graphics, whatever it takes to describe and explain something. Annotation calls out and explains information and, at the same time, explains to viewers how to read data displays. Good annotation is like a knowledgeable expert/teacher at the viewer's side pointing and saying, "Now see how this works with that, how this might explain that..."
Signs seen and unseen
Direct instructions at point of need may encourage writers and programmers to divert diversions. Or not, because signs are seen only a few times before becoming unseen.
What is the strongest visual element?
Preparing to write the novel Catch 22, Joseph Heller composed a storyboard, a 2-dimensional list with 3,650 words arrayed in 34 × 21 = 714 interacting cells. Rows are ordered in time, and each row records when each character does what. Some cell entries are erased. It took 7 years to complete the novel's 758-page typescript.
The Catch 22 plotchart works better upon replacing optically noisy grids with ghost grids. Lightness of framing lines creates soft boundaries to maintain order and also allows words to spill across cells naturally...More generally, ask of information displays and interfaces, "What is the strongest visual element?" The correct answer is not "grid lines".
Your only language is vision
To see with fresh, uninstructed eyes and an open mind requires a deliberate, self-aware act by the observer. Abstract artworks represent themselves and should be first viewed for themselves. When looking at outdoor abstract pieces, concentrate initially on the unique optical experience produced by the artworks. See as the artist saw when making the piece.
A focus on optical experience does not deny stories, it postpones them. Viewing an artwork may evoke interesting narratives – or just tedious artchat recalling similar art or artists, concocting playful tales, realizing how scrap metal was repurposed into art, making judgments about the artist's intentions or character, or contemplating an artwork's provenance, price, politics. Let the artwork stand on its own. Walk around fast and slow, be still, look and see from
up down sideways close afar above below
, enjoy the multiplicity ofsilhouettes shadows dapples clouds airspaces sun earth glowing
. Your only language is vision.Lists consist of whatever it takes
Lists consist of whatever it takes – nouns, proper nouns, verbs, graphics, images, numbers.
...In lists, spaces have meaning, locating elements in relation to other elements. Lists are often free and independent from conventional rules of stylesheets / grammar / typography / punctuation. Lists also help us escape from the personal internalized mash-up stylesheets of every writer and reader – a continuous low-level background buzz checking to see if word usage, spelling, punctuation, grammar are 'correct'. Lists are all content – about the substance contained, not the container. An empirical theory here for reasoning about lists includes
selection of list items
list quality and completeness
comparing list models
comparing list architecturesNo wonder you think it's complicated
We were very proud of our user interface and the fact that we had a way to browse 16,000 (!!) pages of documentation on a CD-ROM. But browsing the hierarchy felt a little complicated to us.
So we asked Tufte to come in and have a look, and were hoping perhaps for a pat on the head or some free advice. He played with our AnswerBook for 90 seconds, turned around, pronounced his review:
"Dr. Spock's Baby and Child Care is a best-selling owner's manual for the most complicated 'product' imaginable – and it has only 2 levels of headings. You have 8 levels of hierarchy and I haven't stopped counting yet. No wonder you think it's complicated."
Verb List
Documents vs. decks
Decks are easier to prepare than documents, however. Documents require coherence, thinking, sentences. But convenience in preparing decks harms the content and the audience. Optimizing presenter convenience is selfish, lazy, and worst of all, replaces thinking.
Books are meant to be used
During most of the course students have their books open, either to read or to follow along. Students are encouraged to annotate the books: "Books are meant to be used. Dog-ear the pages, mark them up, put notes in the spacious margins."
Learning via teaching
The course material changes 15% each year, as the book currently in progress becomes part of the course years before it is finally published. I detect incoherencies and mistakes in the new material while teaching. This leads to refinements or even throwing stuff out from the forthcoming book. A good way to learn about something is to teach it.
Self-publishing, self-exemplifying
I sought to design [my first book] so as to make it self-exemplifying – that is, the physical object itself would reflect the intellectual principles advanced in the book. Publishers seemed appalled at the prospect that an author might govern design. Consequently I decided to self-publish the book.
...[Howard Gralla and I] spent the summer in his studio laying out the book, page by page. We integrated graphics into the text, sometimes in the middle of sentences, eliminating the usual segregation of text and image – one of the ideas Visual Display advocated.
My view on self-publishing was to go all out, to make the best and most elegant and wonderful book possible, without compromise. Otherwise, why do it? The next 4 books were financed by the previous books. I have never written a grant application.
A history of content and sources
Not all that many readers go to the back matter and look up the source for a single sentence. But the back matter can also be read as ordinary text, revealing a history of content and sources. And images and illustrations from the book in the back matter create a lovely visual/verbal summary quilt of the entire book, enjoyed by all.