Jammo the Robot with Unity Sentis (Whisper version)

This is a game demo of a smart robot that can understand player orders and perform them.

You give order using your voice.

To be able to do it, we’re going to use an AI model that transcript voice input and use this input to find the most appropriate NPC action.

The AI model runs directly inside the Unity game thanks to:

• Sentis, the neural network inference library that will allow to run our AI model directly inside our game.
• Sentis-whisper-tiny:  Whisper Tiny model tested to work in Unity 2023. It is a speech-to-text model.
• all-MiniLM-L6-v2: sentence similarity model.

• The Hugging Face Sharp Transformers: a Unity plugin of utilities to run Transformer 🤗 models in Unity games.

You want to create this demo? We wrote a tutorial here 👉 https://www.simoninithomas.com/jammo-sentis-whisper