Research Spotlight: Dr. Dake Zhang

An Interview with Dake Zhang, Associate Professor of Special Education, Department of Educational Psychology 

Can you tell us about a new research project you are working on?

I currently have three active grants focused on developing and evaluating AI-based tools for students struggling with mathematics. For example, I lead an interdisciplinary MathNet project with a Computer Science professor at Rutgers and an educational assessment professor at the University of Washington, Seattle, to develop AI techniques to provide auto-grading and auto-diagnosis to classify students’ problem-solving strategies and error types. We currently focus on students’ hand-written solutions to represent a fraction with a number line, with special attention to the imbalanced data representing the uncommon strategies and error types among students with math difficulties. We obtained 139,000 images related to fractions from the 3.8 million images in the large dataset provided by ASSISTment, an online mathematics educational platform. We have systematically evaluated major Large Language Models (LLMs), including ChatGPT-4o, and found that their computer vision components remain unreliable for interpreting students’ handwritten work, which aligns with other studies on GPT-based grading of handwritten STEM solutions, including visuals. Last year, we used expert-annotated images to train a highly accurate visual processing model. Based on this model, we are developing a “visual translator” that accurately describes and converts key visual features into text. In the following diagnosis step, the textual description serves as input for GPT. This allows GPT to leverage its powerful reasoning capabilities with text input fully.  

In addition to the MathNet project, I have a seed grant in collaboration with math education and computer engineering colleagues to build AI-based platforms offering individualized accommodations and scaffolding for students with mathematical difficulties or disabilities. I am also working with computer science and engineering professors on a grant to design and assess effective visual representations for word problems, including schema-based visual maps aimed at helping college students, particularly those struggling in civil engineering, improve word problem-solving skills.  

Why did you decide to pursue this research? And what is unique about your approach?

My research uses visual representations, such as geometry, number lines, diagrams for word problems, and graphs, to help struggling students solve mathematics problems. My current projects aim to use AI to analyze students’ visual representations in mathematics problem-solving and generate effective visual representations to support students’ problem-solving processes. Incorporating state-of-the-art AI techniques and data science into STEM education empowers educators to answer research questions with a limited sample size that traditional research methods could not answer. It also empowers special educators to provide more individualized support to students with various special needs. My team explores multiple analytic techniques, such as natural language processing to code context characteristics, computer vision to decode students’ visual representations, sequential analysis of action logs, and multimodal machine learning techniques that integrate textual and visual data. While large-language models (e.g., GPT, Gemini, etc.) are already powerful in natural language processing for text inputs, computer vision remains an ongoing challenge in AI. Thus, I successfully attract computer science and engineering collaborators who are eager to drive technological innovations and breakthroughs in computer vision. 

What kind of methodological and theoretical approaches do you use? And why are these important to your work?

I am primarily a quantitative researcher employing a cognitive processing framework. Cognitive processing refers to the mental processes involved in acquiring, processing, and storing information, enabling learning, memory, reasoning, and problem-solving functions. Through the cognitive processing perspective, my research focuses on identifying and understanding barriers and challenges that students with mathematics learning difficulties encounter in mathematical reasoning and problem-solving. For example, many of these students struggle with visual working memory, the ability to temporarily hold and manipulate visual information in mind. Recognizing this issue directs us to develop effective interventions or accommodations, such as creating schematic diagrams that help students identify underlying quantitative patterns, relations, or schemas within geometry or word problems. However, mathematics and special education teachers often lack the expertise to determine suitable cognitive supports or generate supporting tools. Generative AI offers a promising solution, providing opportunities to create mathematics problems with individualized accommodation or scaffolding tools for mass production of classroom instruction, practice, activities, and standard assessments.  

 What’s next for you in terms of research or this project? 

This line of research is engaging and enjoyable. I particularly value collaborating with professors and PhD students from Computer Science and Engineering. We plan to make our data open source; for instance, in the MathNet project, we will package our annotated images, scripts, generated textual descriptions, and the visual translator model together to create a benchmark that will be publicly accessible to other AI researchers. Additionally, we are developing teacher-friendly interfaces for the three grants we are currently working on. We want to attract more funds from research foundations and the industry. In response to Dean Span’s initiative on AI education, I am fortunate to follow the lead of Dean Span and Professor Janice Gobert to develop a large-scale research proposal on AI and STEM education. I sincerely appreciate the opportunity to work alongside and learn from Professor Gobert, who is highly regarded in AI and STEM education.