Callie Y. Kim

Callie Y. Kim

PhD Candidate, Computer Science, University of Wisconsin-Madison

Hi, I am a sixth-year PhD candidate in the People and Robots Laboratory at the Computer Sciences Department of University of Wisconsin-Madison, advised by Bilge Mutlu. I completed my BS in Computer Science and Engineering at Ewha Womans University in 2019 and my MS in Computer Science at the University of Maryland in 2021.

My research in human-robot interaction (HRI) focuses on aligning robot behavior with what people actually want, especially when intent is ambiguous, context-dependent, and hard to fully specify. I design interactive systems that help people specify, verify, and refine robot behavior through three complementary mechanisms: scaffolding that structures how people express intent, verification that makes robot interpretation inspectable and correctable, and proactivity that lets robots anticipate ambiguity, ask for clarification, and learn from users. My work spans extended reality (XR) interfaces, LLM-based agents, and vision-language-action models (VLAs), grounded in empirical studies with end users.

When I’m not working on robots, I enjoy reading books and playing games, especially Role-Playing games (RPG). I also have a background in game development from my undergrad years, which you can check out on my Projects page.

Portrait of Callie Y. Kim
A UR3e robot arm with a two-finger gripper standing on a round platform.
A live UR3e arm, the robot from my RoboCritics study. It picks up a coin and drops it in the box on its own. Switch to interactive to move it yourself.

Publications

Google Scholar

SHRIMP: Iterative Refinement of Robot Task Plans

Mya Schroder, Yuna Hwang, Callie Y. Kim, Leqian Cheng, Jeffrey Li-cheng Liu, Chenchen Zheng, Xinning He, Bilge Mutlu

UIST 2026, ACM Symposium on User Interface Software and Technology

SHRIMP lets users generate hierarchical robot task plans from natural language and iteratively refine them through re-prompting and explicit correction. Users can validate plans in simulation before executing them on a physical robot. A user study with 35 participants showed improved perceived control and robot transparency.

Robo-Blocks: Generative Scaffolding in End-User Design and Programming of Social Robots

Arissa J. Sato*, Callie Y. Kim*, Nathan Thomas White, Abhinav Maneesh, Yuqing Wang, Hui-Ru Ho, Bilge Mutlu (* equal contribution)

DIS 2026, ACM Designing Interactive Systems Conference

Robo-Blocks is a block-based programming environment that leverages LLMs to give novice robot programmers generative scaffolding, connecting high-level ideas to executable robot behaviors through structured narratives. Through a Research through Design process and deployment with novices, we uncovered user personas and usage patterns for generative scaffolding. We present design insights for integrating generative scaffolding into social-robot programming.

Supporting Money Management among Adults with Down Syndrome: A Multi-Technology Probe Study

Hailey L. Johnson, Heidi Spalitta, Callie Y. Kim, Bilge Mutlu

CHI 2026, ACM Conference on Human Factors in Computing Systems

To understand how adults with Down syndrome manage money and what support they prefer, we designed three cash-based budgeting technology probes: a gamified tablet app, an augmented reality app, and a custom tangible device. Seven adults with Down syndrome used all three probes, revealing modality tradeoffs in engagement, verification, and error recovery. We offer design implications for budgeting technologies that support autonomy without collapsing interdependence.

RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics

Callie Y. Kim, Nathan Thomas White, Evan He, Frederic Sala, Bilge Mutlu

HRI 2026, ACM/IEEE International Conference on Human-Robot Interaction Honorable Mention Award

RoboCritics augments LLM-based robot programming with expert-informed motion-level critics that analyze execution traces for issues such as collisions and joint speed violations. The critics surface transparent feedback and one-click fixes that keep users in the loop. In a user study with a UR3e robot, RoboCritics reduced safety violations and improved execution quality compared to a baseline LLM interface.

Elements of Robot Morphology: Supporting Designers in Robot Form Exploration

Amy Koike, Serena Ge Guo, Xinning He, Callie Y. Kim, Dakota Sullivan, Bilge Mutlu

HRI 2026, ACM/IEEE International Conference on Human-Robot Interaction

We introduce Elements of Robot Morphology, a framework identifying five fundamental elements of robot form: perception, articulation, end effectors, locomotion, and structure. We operationalize the framework with Morphology Exploration Blocks, a set of tangible blocks for hands-on, collaborative experimentation with robot morphologies. A case study and design workshops show how they support analysis, ideation, reflection, and collaborative robot design.

Bridging Generations using AI-Supported Co-Creative Activities

Callie Y. Kim, Arissa J. Sato, Nathan Thomas White, Hui-Ru Ho, Christine P. Lee, Yuna Hwang, Bilge Mutlu

CHI 2025, ACM Conference on Human Factors in Computing Systems Honorable Mention Award

We studied how AI-supported story creation can foster intergenerational connection through a user study with 29 grandparent-grandchild groups. Grandchildren managed the technical aspects while grandparents contributed creative ideas, with AI structuring the activity and balancing contributions across generations. We offer design implications for integrating AI into intergenerational co-creative activities.

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

Callie Y. Kim*, Christine P. Lee*, Bilge Mutlu (* equal contribution)

HRI 2024, ACM/IEEE International Conference on Human-Robot Interaction

We compared an LLM-powered social robot against text- and voice-based agents in a user study with 32 participants across conversational tasks including choosing, generating, executing, and negotiating. LLM-powered robots elevated expectations for sophisticated non-verbal cues and excelled in connection-building and deliberation, but fell short in logical communication. We provide design implications for robots integrating LLMs and for fine-tuning LLMs for use with robots.

Research Projects

All projects
  • RoboCritics

    An approach that augments LLM-based robot programming with expert-informed motion-level critics that analyze execution traces for safety violations, surface transparent feedback, and offer one-click fixes.

  • SHRIMP

    System that generates hierarchical robot task plans from natural language and lets users iteratively refine them through re-prompting and explicit correction, validating in simulation before execution.

  • RoboBlocks

    Block-based scaffolding system that helps non-expert users express task intent for robot programming.

  • AR Budgeting Probe

    iPad AR budgeting probe using a CreateML model trained to detect bills from a camera capture. Detected bills populate an editable panel, and instructions are read aloud with Google Text-to-Speech on tap. One of three probes studying how adults with Down syndrome manage money.

  • A robot arm seen through an augmented reality overlay.

    CoframeAR

    Robot Programming Interface in AR for Novice Programmers

  • A hand holding a small silicone component for the headset.

    Custom VR Headset for People with Visual Impairments

    Custom VR headset designed to accommodate people with visual impairments.

Teaching

CS502 Theory and Practice in Computer Science Education

I serve as a Lecturer Student Assistant, where I teach computer science pedagogy and general teaching practices. In this role, I also train peer mentors to develop effective, inclusive mentoring skills.

Grandparents University

I led workshops teaching novel technologies, including robots and AI, to grandparents and grandchildren and designed interactive activities to bridge generational gaps in technology understanding.