Research & Ethnography
What is unspoken
A nested classificatory hierarchy
Unfinished
Record them all
Recommendations for field notes
In Defense of Browsing
The illustrated guide to a Ph.D.
When users never use the features they asked for
An Article by Austin Z. HenleyWe deployed our tool. Almost no one used it.
The handful that did use it, used it once or twice and barely interacted with it. After a few days, zero people were using it.
Why did they tell me they wanted these features?
Why Most Published Research Findings Are False
A Research Paper by John P.A. IoannidisThere is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
How can we develop transformative tools for thought?
A Research Paper by Andy Matuschak & Michael NielsenConventional tech industry product practice will not produce deep enough subject matter insights to create transformative tools for thought.
...The aspiration is for any team serious about making transformative tools for thought. It’s to create a culture that combines the best parts of modern product practice with the best parts of the (very different) modern research culture. You need the insight-through-making loop to operate, whereby deep, original insights about the subject feed back to change and improve the system, and changes to the system result in deep, original insights about the subject.
A hypothesis is a liability
A Research Paper by Itai Yanai & Martin LercherThere is a hidden cost to having a hypothesis. It arises from the relationship between night science and day science, the two very distinct modes of activity in which scientific ideas are generated and tested, respectively [1, 2]. With a hypothesis in hand, the impressive strengths of day science are unleashed, guiding us in designing tests, estimating parameters, and throwing out the hypothesis if it fails the tests. But when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas.
Keep digging
An Article by Ryan SingerThe hardest thing about customer interviews is knowing where to dig. An effective interview is more like a friendly interrogation. We don’t want to learn what customers think about the product, or what they like or dislike — we want to know what happened and how they chose... To get those answers we can’t just ask surface questions, we have to keep digging back behind the answers to find out what really happened.
Fast Path to a Great UX – Increased Exposure Hours
An Article by Jared SpoolAs we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.
The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.
Weighing up UX
An Article by Jeremy KeithMetrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.
Monkeys testing random designs
A Tweet by Jared SpoolA/B testing is an effective approach to use science to design and deliver deeply-frustrating user experiences.
A/B testing without upfront research is just random monkeys testing random designs to see which of those designs do “best” against random criteria.
If drug testing was actually implemented like most A/B tests, you’d give 2 drugs to 2 groups of people and pick the “winner” by whichever group had fewer deaths.
You and Your Research
A Speech by Richard HammingThis talk centered on Hamming's observations and research on the question "Why do so few scientists make significant contributions and so many are forgotten in the long run?"
The Design of Design
What's Wrong With This Model?
A ChapterDesign process models: A summary argument
- A formal design process model is needed, to help organize design work, to aid communication in and about projects, and for teaching.
- Having a visual, geometric representation of a design process model is crucial, for designers are spatial thinkers. They will most easily learn, think about, share, and talk in terms of a model with a clear geometric picture.
- The Rational Model of design occurs naturally to engineers.
- The linear, step-by-step Rational Model is highly misleading. It does not reflect what real designers do, or what the best design thinkers identify as the essence of the design process.
- The bad model matters. It has led to the too-early binding of requirements, leading in turn to bloated products and schedule/budget/performance disasters.
- The Rational Model has persisted in practice despite its inadequacies and plenty of cogent critiques. This is because of its seductive logical simplicity, and because builders and clients needs “contracts."
- Several alternative models have been proposed. I find Boehm’s Spiral Model the most promising. We need to keep developing it.
The spiral model
The spiral shape certainly suggests progress. It associates successive repetitions of the same activity. The geometric shape is easily understood and memorable. The model emphasizes prototyping, starting with user-interface prototypes and user testing long before an operational prototype is possible.
Since a development model is principally used by developers, I believe having it designer-centered is entirely appropriate. With Boehm and against Denning and Dragon, I advocate frequent but not continuous interaction with representative users, with successive prototypes as the vehicles.
I strongly believe that way forward is to embrace and develop the Spiral Model.
A grossly obese set of requirements
Who advocates in the requirements process for the product itself—its conceptual integrity, its efficiency, its economy, it’s robustness? Often, no one. As often, an architect or engineer who can offer only opinion based on taste and instinct, unbuttressed as yet by facts. For in a classical Waterfall Model product process, requirements are set before design is begun.
The result, of course, is a grossly obese set of requirements, the union of many wish lists, assembled without constraints. Usually, the list is neither prioritized nor weighted. The social forces in the committee forbid the painful conflicts occasioned by even weighting, much less prioritizing.
Requirements proliferation
Any attempt to formulate all possible requirements at the start of a project will fail and would cause considerable delays. — Pahl and Beitz, Engineering Design
As Project Manager, I had to reject the requirements document as totally impractical, and have a quite small team of architects, marketers, and implementers extract the essence.
Requirements proliferation must be fought, by both birth control and infanticide.
The architectural contracting model
It is the necessity for contracts, whether within an organization or between organizations, that forces the too-early binding of goals, requirements, constraints. The pressure for a complete and agreed-upon set of requirements run into the hard fact, that it is essentially impossible to specify any complete and accurate set of requirements for any complex system except in iterative interaction with the design process.
How have the centuries-old building design disciplines handled this perplexity? Fundamentally, by a quite different contracting model.
- The client develops a program, not a specification, for the building.
- He contracts with an architect, usually on an hourly or percentage basis, for services, not for a specified product.
- The architect elicits from the client, the users, and other stakeholders a more complete program, which does not pretend to be a rigid contractable product specification.
- The architect does a conceptual design that approximates the reconciliation of program and the constraints of budget, schedule, and code. This serves as a first prototype, to be conceptually tested by the stakeholders.
- After iteration, the architect performs design development, often producing more detailed drawings, a 3-D scale model, mockups, and so on. After stakeholder iteration, the architect produces construction drawings and specifications.
- The client uses these drawings and specifications to enter into a fixed-price contract for the product.
Notice how this long-evolved model separates the contract for design from the contract for construction. Even when both are performed by the same organization, this separation clarifies many things.
The rational model of design
Engineers seem to have a clear, if usually implicit, model of the process of design. It is usually an orderly model of an orderly process as the engineer conceives it.
The notion that the design process should be modeled as a systematic step-by-step process seems to have first developed in the German mechanical engineering community.
Herbert Simon independently argues for design as a search process in The Sciences of the Artificial. He was motivated to lay out a strictly rational model of design precisely because such a model was a necessary precursor to automating design. His model remains influential even if today we recognize the "wicked problem" of original design as one of the least promising candidates for AI.
In software engineering, Winston Royce independently introduced a seven-step Waterfall Model to bring order to the process. In fact, Royce introduced his waterfall as a straw man that he then argued against, but many people have cited and followed the straw man rather than his more sophisticated models. Even if ironically, Royce's seven-step model must be considered one of the foundational statements of the Rational Model of Design.
Design process models
Any systematization of the design process is a great step forward compared to "Let's just start coding, or building." It:
- Provides clear steps for planning a design project
- Furnishes clearly definable milestones
- Suggests project organization and staffing
- Helps communication within the design team
- Is readily teachable to novices, and tells novices facing their first design assignments where to begin.
The Rational Model in particular brings yet more advantages. The early explicit statement of goals, secondary desiderata, and constraints helps a team avoid wandering, and it breeds team unification on purposes. Planning the whole design process before starting coding or formal drawings avoids many troubles and much wasted effort. Casting the process as a systematic search of a design space broadens the horizon of the individual designers and lifts their eyes far beyond their previous personal experiences.
But the rational model is much too simplistic, even in Simon's richly developed version.
The dual ladder
The first task for growing designers, as opposed to managers, is to craft a proper career path for them, one whose compensation and sociological status reflect their true value to the creative enterprise. This is commonly called the dual ladder. It it easy to give corresponding salaries to corresponding rungs, but it requires strong proactive measures to give them equal prestige: equal offices, equal staff support, reverse-biased raises when duties change.
Why does the dual ladder need special attention? Perhaps because managers, being human, are inherently inclined to consider their own tasks more difficult and important than design and need to deliberately assess what makes creativity and innovation happen.
A platonic ideal
As the architecture design progressed, I observed what at first seemed quite strange. For the architecture team, the real System/360 was the Design Concept itself, a Platonic ideal computer. Those physical and electrical Model 50, Model 60, Model 70, and Model 90 things under construction out on the engineering floors were but Plato’s shadows of the real System/360. The real System/360’s most complete and faithful embodiment was not in silicon, copper, and steel, but in the prose and diagrams of IBM System/360 Principles of Operation, the programmer’s machine language manual.
I had a similar experience with the View/360 beach house. Its Design Concept came to be real long before any construction began. It persisted through many versions of drawings and cardboard models.
The design concept
Is there positive value to recognizing an invisible Design Concept as a real entity in design conversations? I think so.
First, great designs have conceptual integrity—unity, economy, clarity. They not only work, they delight, as Vitruvius first articulated. We use terms such as elegant, clean, beautiful to talk about bridges, sonatas, circuits, bicycles, computers, and iPhones. Recognizing the Design Concept as an entity helps us to seek its integrity in our own solo designs, to work together for it in team designs, and to teach it to our youth.
Second, talking frequently about the Design Concept as such vastly aids communication within a design team. Unity of concept is the goal; it is achieved only by much conversation.
Thus, moviemakers use storyboards to keep their design conversations focused on the Design Concept, rather than on implementation details.
The Idea
The design is thus the mental formulation, which Sayers calls “the Idea,” and it can be complete before any realization is begun. Mozart’s response to his father’s inquiry about an opera due to the duke in three weeks both stuns us and clarifies the concept.
For most human makers of things, the incompletenesses and inconsistencies of our ideas become clear only during implementation. Thus it is that writing, experimentation, “working out,” are essential disciplines for the theoretician.
The boldest decisions
In retrospect, many of the case studies have a striking common attribute: the boldest design decisions, whoever made them, have accounted for a high fraction of the goodness of the outcome. These bold decisions were made due sometimes to vision, sometimes to desperation. They were always gambles, requiring extra investment in hopes of getting a much better result.
Intuition and systems
Systematic design excluding intuition yields pedestrian follow-ons and knock-offs; intuitive design without system yields flawed fancies. How to weld intuition and systematic approach? How to grow as a designer? How to function in a design team?