Science & Technology

Prof. Judy Fan: Cognitive Tools for Making the Invisible Visible

by MIT Siegel Family Quest for Intelligence

📚 Main topics

Cognitive tools and visual abstractionJudy Fan studies how people use invented tools—such as number lines, drawings, and graphs—to make information visible, communicate knowledge, and discover useful abstractions. Her broader goal is to explain how people use what they learn about the world to create new things. 3:47
Drawings adapt to context and purposeExperiments found that people make more detailed drawings when they need to distinguish similar objects, and sparser drawings when a broad category is enough. When explaining how a machine works, people emphasize causal parts and motion, while depictions intended for identification preserve more visual detail. 17:57
Measuring human-like visual understandingThe SEVA benchmark compared people and vision algorithms on sketches made under different time constraints. Although some models performed better than others, human recognition remained more reliable, and people and CLIPasso simplified sketches in different ways when production time was limited. 30:05
How people understand data visualizationsFan’s group studies how people interpret graphs, choose plots to answer questions, and learn quantitative reasoning. Their evaluations found that multimodal AI systems still differ from people in both performance and the patterns of their mistakes. 38:04

✨ Key takeaways

Meaning depends on contextA drawing’s meaning is not explained by resemblance alone; people adjust what they depict according to the audience, the task, and shared context. Visual explanations can communicate mechanisms effectively by sacrificing some fidelity to appearance. 17:23
Assess errors, not just scoresSimilar overall accuracy does not mean a model thinks like a person. Studying which mistakes people and models make can help reveal whether a difficulty comes from perception, reasoning, or another step in interpreting a graph. 44:30
People select plots based on their usefulnessParticipants’ choices were best explained by sensitivity to how well each plot helped answer the question, rather than by a simple preference for a particular graph type. 46:09
Current assessments may miss important skillsAnalysis of established graph-understanding tests suggested that common categories, such as plot type or question type, do not fully explain people’s error patterns. Better measures are needed to identify the underlying skills. 51:30

🧠 Lessons learned

Treat representations as purposeful choicesRather than labeling drawings as simply true or false, study what people produce under particular conditions and ask why those representations serve their goals. 57:53
Break complex tasks into smaller stepsUnderstanding a graph may require finding the relevant visual feature, relating it to an axis, and using that information to answer a question. Separating these steps can support better assessment and teaching. 1:04:44
Build visual literacy progressivelyLearning graphs may depend on foundational visual, cognitive, and literacy skills. Fan noted that determining an effective sequence for teaching these representations remains an open question. 1:07:51

🏁 Conclusion/next steps

Develop better models and measuresFan’s group aims to build psychological theories of how people use cognitive tools, investigate the gaps between human and machine behavior, and develop improved ways to assess visualization understanding. 55:15
Support learning and future innovationA better understanding of these tools could help people interpret a complex world and improve education, enabling each generation to build on the knowledge and discoveries of those before it. 55:47

Transcript excerpt

0:04 JOSH TENENBAUM: It's my great pleasure to introduce Judy Fan, who's our colloquium speaker. Judy is one of my favorite cognitive scientists really of any age. I think she's definitely one of the leading-- by any objective standard leading of the younger generation people who are-- have just junior faculty or I don't even know how junior you are, but she's-- been assistant professor at Stanford for a couple of years. I think that's a good estimate. And she's by any means one of the rising or glowing superstars

0:36 of our field. She's won a number of awards-- the Glushko Dissertation Prize, the NSF Career Award. I think you won that. Real-time talk introduction. But none of that really is-- that's not the reason that we invited her. I think of Judy as one of the most creative researchers I know of any stage in any field, and I really mean that deeply. She has a background coming out of neuroscience, and in that

🔒 The full, searchable transcript is available with Pro.

Suggestions