Research & Ethnography
What is unspoken
A nested classificatory hierarchy
Unfinished
Record them all
Recommendations for field notes
In Defense of Browsing
The illustrated guide to a Ph.D.
When users never use the features they asked for
An Article by Austin Z. HenleyWe deployed our tool. Almost no one used it.
The handful that did use it, used it once or twice and barely interacted with it. After a few days, zero people were using it.
Why did they tell me they wanted these features?
Why Most Published Research Findings Are False
A Research Paper by John P.A. IoannidisThere is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
How can we develop transformative tools for thought?
A Research Paper by Andy Matuschak & Michael NielsenConventional tech industry product practice will not produce deep enough subject matter insights to create transformative tools for thought.
...The aspiration is for any team serious about making transformative tools for thought. It’s to create a culture that combines the best parts of modern product practice with the best parts of the (very different) modern research culture. You need the insight-through-making loop to operate, whereby deep, original insights about the subject feed back to change and improve the system, and changes to the system result in deep, original insights about the subject.
A hypothesis is a liability
A Research Paper by Itai Yanai & Martin LercherThere is a hidden cost to having a hypothesis. It arises from the relationship between night science and day science, the two very distinct modes of activity in which scientific ideas are generated and tested, respectively [1, 2]. With a hypothesis in hand, the impressive strengths of day science are unleashed, guiding us in designing tests, estimating parameters, and throwing out the hypothesis if it fails the tests. But when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas.
Keep digging
An Article by Ryan SingerThe hardest thing about customer interviews is knowing where to dig. An effective interview is more like a friendly interrogation. We don’t want to learn what customers think about the product, or what they like or dislike — we want to know what happened and how they chose... To get those answers we can’t just ask surface questions, we have to keep digging back behind the answers to find out what really happened.
Fast Path to a Great UX – Increased Exposure Hours
An Article by Jared SpoolAs we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.
The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.
Weighing up UX
An Article by Jeremy KeithMetrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.
Monkeys testing random designs
A Tweet by Jared SpoolA/B testing is an effective approach to use science to design and deliver deeply-frustrating user experiences.
A/B testing without upfront research is just random monkeys testing random designs to see which of those designs do “best” against random criteria.
If drug testing was actually implemented like most A/B tests, you’d give 2 drugs to 2 groups of people and pick the “winner” by whichever group had fewer deaths.
You and Your Research
A Speech by Richard HammingThis talk centered on Hamming's observations and research on the question "Why do so few scientists make significant contributions and so many are forgotten in the long run?"
Winning by Design: The Methods of Gordon Murray
A case study of the working methods of one particularly successful designer in a highly competitive design domain - Formula One racing car design. Gordon Murray was chief designer for the very successful Brabham and McLaren racing car teams in the 1970s and 1980s. His record of success is characterised by innovative breakthroughs, often arising as sudden illuminations, based on considering the task from first principles and from a systemic viewpoint. His working methods are highly personal, and include intensive use of drawings. Personality factors and team management abilities also appear to be relevant. There are some evident similarities with some other successful, innovative designers
You need to make the step forward
Throughout a racing season there is constant, relentless pressure on the designer to keep making design improvements. But there is a limit to what can be achieved with any car design, before a jump has to be made to basically a new design, an innovation. As Gordon Murray says, ‘Given the situation and the pressure at any one time, you do get to the brick wall...I mean you're doing all these normal modifications, you know you can't go any quicker, you need to make the step forward.’
In the midst of the pressure, the fervour, the panic, he ‘used to get breakthroughs, I mean I used to get like suddenly a mental block's lifted.’
Drawing the bits
That's what is great about race car design, because even though you've had the big idea - the “light bulb” thing, which is fun - the real fun is actually taking these individual things, that nobody's every done before, and in no time at all try and think of a way of designing them. And not only think of a way of doing them, but drawing the bits, having them made and testing them.
Like designing things for the first time
Gordon Murray insists on keeping experience 'at the back of your mind, not the front' and to work from first principles when designing. For instance, in designing a component such as a suspension wishbone, 'it's all too easy - and the longer you're in design the easier it is - to say, I know all about wishbones, this is how it's going to look because that's what wishbones look like.' But if you want to make a step forward, if you're looking for ways of making it much better and much lighter, than you have to go right back to load path analysis. It is like designing things for the first time, rather than the nth time.
Wonder Plots
Working from first principles, and working in a highly organized way seem to come naturally to him, but his personal design process is much less structured than the results might suggest. Although he can tightly organize his team and run a complex racing organisation, his personal ways of designing are relatively unstructured, based on annotated, thumb-nail sketches. ‘I don't sit down and say, OK, now I've had the idea, let's see, this is a solution, these are the different ways to go, if I do this, and do that; I do lots of scribbles just to save it, before I forget.’
Gordon’s design process is based on starting with a quick sketch of a whole idea, which is then developed through many different refinements. ‘I do a quick sketch of the whole idea, and then if there's one bit that looks good, instead of rubbing other bits out, I'd put that bit to one side; I'd do it again and expand on the good bit, and drop out the bad bit, and keep doing it, doing it; and end up with all these sketches, and eventually you end up throwing ninety percent of these away.’ He also talks to himself - or rather, writes notes to himself on the sketches; notes such as ‘rubbish’, ‘too heavy’ or ‘move it this way 30mm.’ Eventually he gets to the stage of more formal, orthographic drawings, but still drawing annotated plans, elevations and sections all together, ‘Until at the end of the day the guys at Brabham used to call them “Wonder Plots”, because they used to say “It's a wonder anybody could see what was on them”!’
I never have engineers that aren't designers
Although Gordon Murray carried immense personal responsibility for the design work of his racing cars, inevitably it involved a lot of teamwork. Clearly he has been successful in inspiring others to work with him. He likes to involve team members in the design problems, and for that reason prefers to recruit all-rounders to his team; ‘I never have engineers that aren't designers.’
The problem with CAD
He also likes to work collectively, standing around a drawing board discussing problems and trying ideas.
For this kind of teamwork, and especially for conceptual design work, he finds computer aided design systems too restrictive. For the McLaren F1 super-car, he installed a five-metre long drawing board in the design office, so that the car could be drawn full size. ‘The problem with CAD for this sort of stuff is that you can never have a full-size drawing, unless you do a print, and by the time you do a print it's out of date in the concept stage.’ He also does not like the one-person emphasis of CAD screens; ‘You can only ever talk to one person at once - you stand behind and look over somebody's shoulder, which is not very good for a boss-designer relationship anyway, to have somebody standing behind you is never a good thing. To look over somebody's shoulder at a tiny little screen, it's just wrong, it's totally wrong.’
(On the other hand, he fully acknowledges that tasks like a complex suspension plot to determine the wheel envelope are ideal for CAD.)
Drawing as a means of thinking
Two-dimensional plans or sections can be seen with sketches and more diagrammatic marks all on the same piece of paper in what appears a confusing jumble.’ These sound like Gordon’s ‘wonder plots’. The architects also use their drawings as a means of thinking ‘aloud’, or ‘talking to themselves’, as Gordon put it. For example, Lawson reports the architect Richard MacCormac as saying, ‘I use drawing as a process of criticism and discovery’; and the engineer-architect Santiago Calatrava as saying, ‘To start with you see the thing in your mind and it doesn’t exist on paper and then you start making simple sketches and organizing things and then you start doing layer after layer.... it is very much a dialogue.’
The common elements in these similar descriptions are the use of drawing not only as a means of externalising cognitive images but also of actively ‘thinking by drawing’, and of responding, layer after layer and view after view, to the design as it emerges in the drawings. These observations also confirm Schön’s observation of designing as a ‘reflective conversation’ between the designer and the emerging design. It is the reliance on drawing, and the preference for the immediacy of the interaction and feedback that manual drawing gives, that makes the architects, like Gordon Murray, unenthusiastic about CAD as a conceptual design tool.
A new gestalt
The innovator has a systems mind, one that sees things in terms of how they relate to each other in producing a result, a new gestalt that to some degree changes the world.
Intense activity, then relaxation
The working style is based on periods of intense activity, coupled with other periods of more relaxed, reflective contemplation. This working style may not be a reflection of a particular personality trait, but a necessary aspect of creative work, which requires alternating intense effort with relaxation.
Strategic, not tactical
The working methods of the innovative designer are, for the most part, not systematic; there is little or no evidence of the use of systematic methods of creative thinking, for example. The innovative designer seems to be too involved with the urgent necessity of problem solving to want, or to need, to stand back and consider their working methods. Their design approach is strategic, not tactical.
Drawing for parallel design thinking
An important feature of their strategy is parallel working - keeping design activity going at many levels simultaneously. The best cognitive aid for supporting and maintaining parallel design thinking is drawing. Drawing with the conventional tools of paper and pencil gives the flexibility to shift levels of detail instantaneously; allows partial, different views at different levels of detail to be developed side by side, or above and below and overlapping; keeps records of previous views, ideas and notes that can be accessed relatively quickly and inserted into the current frame of reference; permits and encourages the simultaneous, non-hierarchical participation of co-workers, using a common representation.
The drawing of partial solutions or representations also aids the designer’s thinking processes, and provides some ‘talk-back’. As well as drawing, innovative designers frequently like to undertake practical work related to the design solution, such as building models or mock-ups, or participating in construction.
A small team of committed coworkers
The innovative designer also likes, perhaps needs, to work with a small team of committed co-workers who share the same passions and dedication.