Research & Ethnography
What is unspoken
A nested classificatory hierarchy
Unfinished
Record them all
Recommendations for field notes
In Defense of Browsing
The illustrated guide to a Ph.D.
When users never use the features they asked for
An Article by Austin Z. HenleyWe deployed our tool. Almost no one used it.
The handful that did use it, used it once or twice and barely interacted with it. After a few days, zero people were using it.
Why did they tell me they wanted these features?
Why Most Published Research Findings Are False
A Research Paper by John P.A. IoannidisThere is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
How can we develop transformative tools for thought?
A Research Paper by Andy Matuschak & Michael NielsenConventional tech industry product practice will not produce deep enough subject matter insights to create transformative tools for thought.
...The aspiration is for any team serious about making transformative tools for thought. It’s to create a culture that combines the best parts of modern product practice with the best parts of the (very different) modern research culture. You need the insight-through-making loop to operate, whereby deep, original insights about the subject feed back to change and improve the system, and changes to the system result in deep, original insights about the subject.
A hypothesis is a liability
A Research Paper by Itai Yanai & Martin LercherThere is a hidden cost to having a hypothesis. It arises from the relationship between night science and day science, the two very distinct modes of activity in which scientific ideas are generated and tested, respectively [1, 2]. With a hypothesis in hand, the impressive strengths of day science are unleashed, guiding us in designing tests, estimating parameters, and throwing out the hypothesis if it fails the tests. But when we analyze the results of an experiment, our mental focus on a specific hypothesis can prevent us from exploring other aspects of the data, effectively blinding us to new ideas.
Keep digging
An Article by Ryan SingerThe hardest thing about customer interviews is knowing where to dig. An effective interview is more like a friendly interrogation. We don’t want to learn what customers think about the product, or what they like or dislike — we want to know what happened and how they chose... To get those answers we can’t just ask surface questions, we have to keep digging back behind the answers to find out what really happened.
Fast Path to a Great UX – Increased Exposure Hours
An Article by Jared SpoolAs we’ve been researching what design teams need to do to create great user experiences, we’ve stumbled across an interesting finding. It’s the closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.
The solution? Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.
Weighing up UX
An Article by Jeremy KeithMetrics come up when we’re talking about A/B testing, growth design, and all of the practices that help designers get their seat at the table (to use the well-worn cliché). But while metrics are very useful for measuring design’s benefit to the business, they’re not really cut out for measuring user experience.
Monkeys testing random designs
A Tweet by Jared SpoolA/B testing is an effective approach to use science to design and deliver deeply-frustrating user experiences.
A/B testing without upfront research is just random monkeys testing random designs to see which of those designs do “best” against random criteria.
If drug testing was actually implemented like most A/B tests, you’d give 2 drugs to 2 groups of people and pick the “winner” by whichever group had fewer deaths.
You and Your Research
A Speech by Richard HammingThis talk centered on Hamming's observations and research on the question "Why do so few scientists make significant contributions and so many are forgotten in the long run?"
Understanding Understanding
A dot went for a walk
A dot went for a walk and turned into a line.
The dot, the line, the dance, the story, and the painting had found connections. Memory became learning, learning became understanding.Learning is remembering what one is interested in. Learning, interest, and memory are the tango of understanding.
Creating a map of meaning between data and understanding is the transformation of big data into big understanding.
The dot had embraced understanding.
Understanding precedes action.
Each of us is a dot on a journey.Admitting ignorance
The most essential prerequisite to understanding is to be able to admit when you don't know something. Striving to be the dumbest person in the room.
When you don’t have to filter your inquisitiveness through a smoke screen of intellectual posturing, you can genuinely receive or listen to new information. If you are always trying to disguise your ignorance of a subject, you will be distracted from understanding it.
Information imposters
Information imposters: This is nonsense that masquerades as information because it is postured in the form of information. We automatically give a certain weight to data based on the form in which it is delivered to us. Because we don’t take the time to question this, we assume that we have received some information.
My favorite example of this is in cookbook recipes that call for you to “cook until done.” This doesn’t tell you very much. Why bother? Information imposters are fodder for administravitis.
Michaelangelo's hammer
A young man named Michelangelo stands in front of a huge granite monolith. He stands there at a time in history before the technologies that brought us the hammer and chisel have occurred. He gazes at the rock. He dreams his dream and the best that he is able to say is, What a wonderful stone you are.
…
Michaelangelo now stands in front of the same rock. Thrust into his hands are a hammer in one and a chisel in the other. He looks at his hands, at the technological tools that they hold, and gazing at the same stone, with epiphanic zeal, says I must let Moses out.
I won't get
If I don't ask, I won't get.
Talking with Clinton
When you are talking with Clinton, he is not looking over your shoulder to see who else is in the room. You can tell he is not thinking about how he is going to respond to you. He is there, present and listening.
By the way, when you scramble the letters in the word listen, it becomes a new word: silent. We’re so often wrapped up in our own self-talk, we forget to listen and learn the information in the first place…and you can’t remember or understand something you never observed.
Scales of change
To understand revolutionary change in its full complexity – not just what happened, but why it happened – we need a model that works across multiple scales, and the disciplines that traverse them.
The method
Well no, see, that’s the tricky part. I always try to come up with things that when they find out the method, the method is as interesting as the effect itself. — David Blaine
Selling your own ignorance
When you sell your expertise - and what I mean by sell is to move ahead in a corporation, or sell an idea to a publisher, or sell an ability to a client - by definition, you’re selling from a limited repertoire.
However, when you sell your ignorance to move ahead, when you sell your desire to create and explore and navigate paths to knowledge, when you sell your curiosity - you sell from a bucket with an infinitely deep bottom that represents an unlimited repertoire. And, you sell in a way that’s not intimidating, in a way that joins the explanation to the fascination that comes with understanding.
You only understand something relative to something you already understand
A good question is better than a brilliant answer
The eyes of a traveler
We’ve all heard that travel broadens the mind. But beneath this cliché lies a deep truth. Things stand out because they’re different, so we notice every detail, from street signs to mailboxes to two you pay at a restaurant. We learn a lot when we travel, not because we are any smarter on the road but because we pay such close attention. On a trip, we become our own version of Sherlock Holmes, intensely observing the environment around us. We are continuously trying to figure out a world that is foreign and new.
Too often, we go through our day-to-day life on cruise control, oblivious to huge swaths of our surroundings. To notice friction points – and therefore opportunities to do things better – it helps to see the world with fresh eyes. When you meet creative people with lots of ideas constantly bubbling to the surface, you often come away feeling that they are operating on a different frequency. And they are, most of the time. They have all their receptors on — and frequently turned up to eleven. But the fact is, we are all capable of this mode. Try to engage a beginner’s mind. For kids, everything is novel, so they ask lots of questions, and look at the world wide-eyed, soaking it all in. Everywhere they turn, they tend to think, Isn’t that interesting? rather than, I already know that.
By adopting the eyes of a traveler and a beginner’s mindset, you will notice a lot of details that you might normally have overlook. You put aside assumptions and are fully immersed in the world around you. In this receptive mode, you’re ready to start actively searching out inspiration.
Cart before the horse
It seemed to him that in pursuit of an all-purpose, all-serving health care plan, the country was designing an ocean liner before it even understood why boats float.
The What and the How
How many aspects of problem solving and design solutions are there? Many, many might answer.
But there are actually just two: What to do and how to do it.
Before anything else, before you consider how you’re going to get it done or how much you’ll have to pay for it or how you’ll get permission to do it — before anything else, you have to decide what you want. You have to decide what kind of place you want to live in, what kind of community, what kind of city.
We don’t pay enough attention to the old adage: Be careful what you wish for, because you just may get it.
Thinking with the hand
The idea of thinking with your hands is not the exclusive domain of sculptors or engineers. In healthcare, for exams, surgeons have a powerful link between the head and the hands. Of course, like all healthcare professionals, their work involves a lot of deep knowledge and understanding. But it’s not all in their brain. If you were in medical school, learning to be a surgeon, for example, would you want your professor to be someone who writes about surgery or would you rather be taught by a surgeon? Many surgeons don’t publish scholarly articles. They spent time perfecting their skill with their hands. In some ways, the relationship between surgery and the larger field of medicine is analogous to the relationship between design and the larger field of engineering. Designers — and design thinkers —often think with their hands.
A pattern of understandings
The clock has no finished design and it has been made without any careful drawings or mathematical calculations. The pieces are only made if I can hold their details in my head as I make them, without reference to any set of external measures. I do make rough sketches of some parts as a path to understanding them, but never use these during the making of the parts. The clock gradually grows through trial and error and lots of physical work with metal, but out of this has come a set of principles of making that were not clear to me before doing the clock. I have finally realized that what I am actually making is a pattern of understandings of the process of making rather than the things that are actually being made. — Richard Benson
#15
How different am I, making clock number 15, from the process of natural selection laboring under changing conditions to generate the biological constructs? That ancient evolutionary system works on the basis of trial and error repeated in huge numbers over immense spans of time, with the failures discarded and the successes retained. At times it seems to me that my clock making is quite similar, as my mind, just barely thinking, sorts through huge numbers of possibilities and discards them as failures before even trying them, so the few that are made have a pretty good chance of success. Is this foresight some form of understanding? I think not. No revelation here, just enough thinking to spur the maker on to cut some piece of metal which, once made, might fail or succeed. Yet — in either case — the thing made and its creation remains the sole root of any real understanding that takes place. The clock is crude but gets built, and even in its base simplicity teaches its maker how to understand what must be understood for something to be made. — Richard Benson
A classroom without walls
The city is education – and the architecture of education rarely has much to do with the building of schools. The city is a schoolhouse, and its ground floor is both bulletin board and library. The graffiti of the city are its window displays announcing themselves, telling about what they’re doing and why and where they’re doing it. Everything we do – if described, made clear, and made observable – is education: the Show and Tell, the city itself. It is a classroom without walls, an open university for people of all ages offering a boundless curriculum with unlimited expertise. If we can make our urban environment comprehensible and observable, we will have created classrooms with endless windows on the world.