AI Psychological Counseling System

Signals beyond words

I wanted to work with reactions that the content of a conversation alone does not reveal. This prototype ties webcam facial expression recognition, voice conversation, and an Arduino heart-rate sensor into a single counseling session.

The session runs in sequence. First the camera reads the user’s expression for a few seconds. Coordinate rules on facial landmarks, such as the corners of the mouth and how far the mouth is open, classify it into one of four states. Then, when the user speaks, Korean speech recognition transcribes it, and the expression just read, the utterance, and the conversation history so far are passed together to a language model. The response comes back through speech synthesis. The prompt forces every counseling response to mention the user’s expression, so the user can feel that what is observed is reflected in the conversation.

Three inputs meet in the report

When the conversation ends, the Arduino collects heart rate and reads temperature and humidity. The conversation history, the expression at each turn, and the biosignals are then summarized in a single prompt and saved as a PDF report with a heart-rate graph. The three inputs meet on one record after the session ends.

The substance of this work was sequencing: placing inputs that arrive at different speeds within one session flow.

This work does not provide medical diagnosis or professional counseling. It is a prototype to test whether a conversation experience that responds to several sensory inputs is possible.