Shan Gao

Shan Gao

Ph.D. Student in Computational Cognitive Neuroscience

A (Speculative 😃) AI Coach for Storytelling

3 minute read

Published:

Why

Storytelling is an essential skill in almost every aspect of our life, from school, work, to all kinds of relationships:

  • Interviewing? Tell a good story about that career change to persuade the interviewer that you are the perfict fit for them;
  • Reporting scientific findings? Tell a good story about how your data connects to the big-picture questions your community cares about to get them interested;
  • Having to reschedule an important meeting a second time due to an emergency? Tell a good story to convey the emergency and convince other attendees that you value their time;
  • Leading a team? Tell a good story about how everyone will be able to grow through this project to keep them motivated;
  • …

In each conversation (broadly defined!), we need to tell a good story about what we have in mind in order to get our audience convinced, impressed, or simply engaged enough to keep the conversation going.

However, telling a good story can be difficult. Among the many cognitive processes involved, keeping track of the messages we’d like to get across is just the beginning. More importantly, we need to infer our audience’s mental state changes in real time – inferring how THEY infer (i.e., decode) our own mental states (i.e., the messages they THINK we would like to convey) given how we express (i.e., encode) the messages – and design subsequent narratives accordingly to elicit the effects we’d like to achieve. (This paper on “Storytelling as Inverse Inverse Planning” by Chandra et al. provides more background.) The amount of theory of mind (ToM) reasoning required makes storytelling challenging, especially for neural divergent populations and/or people lacking the opportunity to practice it growing up. Additionally, there are obviously differences in what consititue a “good story” across cultures and situations; e.g., sometimes a straighforward framing is helpful, while other times a more tactful approach is appreciated.

Human coaches for storytelling can be hard to access due to cost, scheduling, etc. So I designed a mobile app UI prototype of a (speculative) AI coach for storytelling – “HiStory” – to complement traditional coaching:
* Embedded prototype may not be displayed properly in mobile browsers due to scaling; for best results, request desktop site.

What

HiStory is designed to be an AI-powered video conversation simulator and analyzer that provides personalized “face-to-face” (yes, simulated) practice sessions with real time feedback.

  • Video conversation simulation: Based on text and video generative AI tools such as ChatGPT and Soul Machines.
  • Real time conversation analysis: Evaluating user’s storytelling through explicit (e.g., contents and language use like in this paper) and implicit (e.g., through facial emotion recognition) information. Results will also be used to simulate downstream responses of the AI coach (i.e., audience).
  • “Mind-reading” mode: Explicitly displays audience’s mental state changes in response to user’s storytelling (generated based on real time conversation analysis) to aid practice.
  • Retention features: Assessment and progress tracking; daily challenge and engagement graph.

The prototype can be adapted to related domains, such as AI-powered therapy for neurodivergent populations complementing traditional therapy sessions.

UI Libraries & GenAI Used