Hacker News

MrRowTheBoat
Show HN: Voice driven murder mystery, Interview AI suspects with your voice whodunnitai.com

Hey HN!

I'm excited to show off this really fun project I put together. I originally built this project 2-3 years ago, AI was already booming at the time, however voice AI agents were still very early. I loved my proof of concept at the time, but wasn't quite happy with it.

I recently had the desire to check out the tech again, and know many of you will be interested.

Interviews are speech to speech with OpenAI's gpt-realtime-2.1 over WebRTC. This model is... expensive, and because of that, I have to add some amount of restrictions, conversations are tied to a authenticated Clerk user id. I have also added a 30 minute timer because well, I really don't want to go broke while I sleep tonight.

Each suspect has a tool they call when you make a direct accusation. It captures who you accused and a faithful list of the evidence you actually stated.

A separate gpt-5-mini judge then decides which of the case's required evidence facts you genuinely presented. Paraphrasing counts, vague suspicion and fishing don't.

The rest is Next.js, MongoDB, and Clerk.

Let me know whether the suspects hold up under a real interrogation.


MrRowTheBoatop4 days ago

Hi everyone!!! Thanks so much for the support.

I launched this at midnight prior to logging off and going to bed, funnily enough a massive storm hit us around 3am and I woke up and noticed my OpenAI account failed to fund lol. So we ran out of funds and it caused things to break for some of you.

Just waking up here on the East coast, once my daughter is occupied I will try to answer more of your comments!

MrRowTheBoatop4 days ago

We are at $100 usage since posting this haha. I will probably cap it at $500 for you guys

dmd4 days ago

It would be nice to be able to play with one's own account, so it's not billing you.

I know there's a way to do this with A\ powered apps like this, I'm sure there must be for OAI?

(By the way, you're exposing the key. So anyone can just use up that key for their own nefarious purposes...)

MrRowTheBoatop4 days ago

Yes!! This was a plan I had, BYOK, Bring your own key

I may do this today

cs19964 days ago

Instead of that, I would not put in my own key, you should get a brand partnership. Something like the Liquid Death Murder Your Thirst water brand. https://liquiddeath.com This would be perfect for them and paying for all the AI use would be nothing compared to their normal advertising. Win win.

MrRowTheBoatop4 days ago

Are you proposing a small add on the site? or does the AI agent do something like this:

"Hey give me a second to answer while I take a sip of this crispy liquid death, these killer cans of soda-style sparkling will attack your tastebuds with soda-level flavor and only 1/10th of the sugar."

sphars4 days ago

"Drink verification can to continue"

MrRowTheBoatop4 days ago

"Please slurp as loud as possible into the mic"

andrewfromx4 days ago

[flagged]

ramesh314 days ago

Maybe play around with CPU inference on smaller models too. Frontier models are probably overkill for simple language responses at this point; I suspect you could get pretty good results with Qwen doing this, and Kokoro TTS is really good now.

senordevnyc4 days ago

Dude, just wanted to be sure you saw his comment about exposing your API key. You do not want to do that!

MrRowTheBoatop4 days ago

I think I replied to this, ephemeral secrets are minted per browser

panorama4 days ago

Very neat idea, and overall it is more well-done than I assumed a vibe-coded game would be. Unfortunately I ran out of time and now the game locks me out of doing anything and I don't know who the murderer is :( I wish there was just a way to have it revealed.

I've always thought that a game like Civilization should allow the user to interact with different opposing world leaders, but in a non-deterministic way as opposed to a subset of fixed interactions. Given what you know about this leader's temperament, goals, and so on, you can conduct diplomacy in a way that can be way more immersive and crafty than what the game currently gives you, which hasn't seem to evolve much in decades.

WillAdams4 days ago

conception3 days ago

Diplomacy is great but computationally suffers from deterministic outcomes - some starts are better than others, there are actual optimal moves to make, you can (and should) play to a stalemate, etc. which take away from the fun - playing against the best moves to make vs against a crafty person.

snarf213 days ago

There is a new Diplomacy card game (The Golden Blade) that is that is shorter and more approachable (plus no player elimination) and not solvable. You still play secret orders; it is just that now your "borders" are your left and right opponent. You can still work together for extra benefit or betray your neighbor and backstab them. Obviously, there is a lot less potential for complex tactics each turn but it still gives some of the vibes of the classic in a format that is easy to get to the table. There are a lot of YT videos from content creators that you can get a sense of how it plays.

Source: I designed it :) (sorry for the minor self promotion)

conception3 days ago

I’ll check it out!

MrRowTheBoatop4 days ago

sorry :( I will think about this some more, the only reason I have to lock people out is because I can't afford the current costs lol

twosdai4 days ago

Byok or monetize it. Seems like a great game. :)

MrRowTheBoatop3 days ago

DONE

smusamashah4 days ago

Please add a demo video/voice where you show someone asking questions and AI answering it. I wanted to see what it is like but dont want to signup.

MrRowTheBoatop4 days ago

I will do this, I was not sure how to do this cleanly, my recording I did was a little awkward haha

arexxbifs4 days ago

I played various murder mystery games this summer and one of them had this feature: you could call a suspect and ask arbitrary questions for a limited amount of time (three minutes or something). The problem was of course that the model hallucinated an answer to one of the questions, suggesting that other suspects were related by blood even though they clearly weren't.

ChrisGreenHeur4 days ago

So it was very life like and realistic.

arexxbifs4 days ago

Not really, since it was something that particular character would clearly have known about and wouldn't have gained anything from lying about.

ChrisGreenHeur4 days ago

Again, sounds very realistic.

MrRowTheBoatop4 days ago

This is solved by having a shared knowledge base between the agents!

b800h4 days ago

Even the tag-line, "one poisoned patriarch, one locked-in truth" sounds very AI, and makes me think that I'd struggle to enjoy this. But I'll give it a go, because the idea is excellent.

I think this would work very very well in combination with human writers.

MrRowTheBoatop4 days ago

Hiiiii, i would appreciate your feedback now that I resolved the openai issue. I think you will be surprised. I was!

JoeDaDude4 days ago

I have an LLM running locally on my machine complete with STT and TTS. While I don't expect much, it will be fun to get my LLM to play the game.

MrRowTheBoatop4 days ago

hahaha, omg. I'd love to see a video of this

wfvr4 days ago

Congrats, I have not had the chance to try it yet, but what I saw looked awesome and I'll give it a go, for sure. I'll update this comment when I do.

I've been working on a similar concept for 6 months now (https://game.selna.ai, portuguese only for now, sorry!), and it's been a journey. I've been working with VR professionally for some 8 years now and creating what I call a "social virtual reality" seemed simple enough, but has turned out to be much more complex than I initially imagined.

In the same way that in normal VR a single badly scaled object can break immersion, in "social VR" a single contradiction kills the experience. But when stars align and eveything works, it's magical.

I started with whodunnits but have expanded it into linear and non-linear storytelling. At this point the generation pipeline has some 40+ separate steps, the output of ones feeding others, until it all comes down to single prompt the NPC uses to talk using the realtime API. The platform can create a complete narrative (including cast personalities and images) in some 20 minutes, and in the process generates the equivalent to a big book of text.

I've been working on this just because I can and I am a bit obsessed with getting it working perfectly, and I don't really know what I'll do with it when it does.

wfvr4 days ago

Sorry for the self-reply, I couldn't figure out how to edit the comment. I tried it now, it's very good. I love the fact that you can just move from one suspect to the other quickly.

You'll need to generate and give the characters a canonical current date and time, that will fix a lot of different subtle issues.

You have a huge break of character or the fourth wall whenever the player makes an accusation, the character goes like "you're making a serious claim, let me formally validate it before I respond" and then they refuse to reply to anything else while "the response is being validated".

MrRowTheBoatop4 days ago

Love it, and I think this is the future for RPG gaming. Endless immersion. I see this project expanding into "experiences" or "scenarios" as well, but I like to focus on small things first. If people like this, I will build more. If it's a cool demo, but not worth putting more time into, I'll probably just move on.

grahamburger4 days ago

Neat! I've been wanting to make something like this for a long time, well before LLMs came around. It was one of the first things I tried out with an LLM in the early days of ChatGPT. My son and I made little mysteries for each other and just pasted them in a chatgpt conversation with instructions for it to pretend to be the other characters and whatnot, then handed the keyboard to the player. Mine didn't work that well, I don't think he ever solved it; partially because chatgpt kinda led him astray. When I started his, I just point blank asked the mystery and it told me the solution. I felt bad about that, he had put a good amount of effort into it, but I had reminded him to tell it not to reveal the answer and he didn't. Anyway, I figured someone would make this eventually! Excited to give it a try!

MrRowTheBoatop3 days ago

In an effort to keep things running longer, before hitting my budget of $500, I am reducing times to 15 minutes, and I've added BYOK.

I don't think I can make everyone happy with the solution, your key is saved on your own browsers localStorage, I do not save or store it in my database. I suggest everyone creates a new key dedicated to our service so you can invalidate it later on. That key is sent to our api using `x-openai-key` header and then we mint ephemeral client secrets to perform web rtc functions.

MrRowTheBoatop3 days ago

I also added a donation button, no donation is required, but I have had a few people request this to assist with the costs.

ethannam90224 days ago

This is a really fun take on a detective game. One small thing I noticed: during the initial detective briefing, the time always seems to show 0:00. Not sure if that’s intentional or a bug.

MrRowTheBoatop4 days ago

Oops, ran out of OpenAI credits

garbagewoman4 days ago

[flagged]

bartleeanderson4 days ago

sounds good, but it won't get past Dispatch confirmation. You should specify what networking permissions or whatever is needed. My It goes red and the text to read disappears.

MrRowTheBoatop4 days ago

OpenAI credits ran out! Resolved!

b800h4 days ago

Same.

camillomiller4 days ago

Vibecoding at its finest

BenzeneDream4 days ago

Useless comment of the day

joe-dakroub3 days ago

Really like this idea and the way you implemented it is very interesting to read about. BYOK is definitely needed as people will want to use this with their own account. Genuinely novel and nicely done. Keep it up and thank you!

lorenzo954 days ago

sounds neat, would love to try it but I don't make accounts for websites anymore.

MrRowTheBoatop4 days ago

Very understandable… I originally didn’t have accounts when sharing it to friends. The account is really just there as a barrier to try and circumvent bots and bad actors as much as possible, even though there are still flaws. The goal was simply to have a way to force a max timer on someone so that more people can try this out

garbagewoman4 days ago

so if you got locked out of hn, you would just never comment again?

what a strange comment

yoz-y4 days ago

I would assume it implies “websites that have no need to actually have an account”.

Although most time I cave in, I really understand the sentiment of the parent.

watwut4 days ago

It makes perfect sense. You are being intentionally obtuse.

garbagewoman4 days ago

setting an arbitrary point in time where you no longer will sign up for accounts does not, in fact, make perfect sense to me.

aljgz4 days ago

[flagged]

ShinyLeftPad4 days ago

Good way to get more people's voices into training dataset

ryanjshaw4 days ago

Is getting people’s voices difficult? Doesn’t YT have more than enough data already?

MrRowTheBoatop4 days ago

I have no idea how to do that, but yes, as someone else noted, OpenAI might!

ShinyLeftPad4 days ago

You can assess if model provider gets any identifiable data about the participating person though. Is that the case?

MrRowTheBoatop4 days ago

The authentication service, Clerk, will gain your First name, and Email address if you use Google SSO.

The model provider does not have any identifiable data such as emails, user ids, first name, nothing at all related to your User account. it's the agent context, and the user who is identified in prompt as "DETECTIVE". I don't know if this answers your question.

ShinyLeftPad4 days ago

I think it does thanks!

amelius4 days ago

Maybe run a STT -> TTS filter in front of your mic?

MrRowTheBoatop4 days ago

I've built this pipeline before on my first iteration, because it's cheaper, it dooes add a lot of latency :(

ShinyLeftPad4 days ago

I guess could work, real mic -> stt -> tts -> emit from a virtual audio device configured as the mic...

amelius4 days ago

I only hope latency is not a problem.

wartywhoa234 days ago

The only relevant comment!

qwertox4 days ago

Not really. It is true in a way, but the question is if this game is made to collect data, or just to present a game, where the recordings are then deleted, in good faith.

The developer's homepage gives the impression that this is more for the tech and fun aspect, rather than collecting speech samples.

ShinyLeftPad4 days ago

this uses a proprietary gpt model and so the data is sent to closedai. even if the developer is in good faith you can expect your voice to be part of training data.

[deleted]4 days agocollapsed

fillok56862 days ago

Out of curiosity, how do you handle prompt injection(user cheats) for a secret leak?

MrRowTheBoatop2 days ago

Haha. I don’t! Lol. Was the curiosity based on a practical experience with the voice ai? Or mostly curiosity if I did anything to address it?

I noticed it does ignore “ignore previous instructions” but my guess is this is baked into open ai now

kunaaldhawan13 days ago

Sounds wild! Just a question though, are you planning to take this to legal level to help Govts and police?

MrRowTheBoatop3 days ago

Uhhm, I’m not sure what you mean? Like help them train?

donpdonp4 days ago

even without trying it, its so cool this exists. I think this kind of voice interaction has been a dream of adventure game creators for so long, and now its possible. there was a game many years ago (in the days of messengers like trillium being popular), I forget the name, where one of the characters would make a landline phone call to you. The voice acting was great and it felt novel and exciting to see it work.

Harvya day ago

I've just played it to the end with my own OAI key. Only $4 to solve which I thought was value for money.

A few issues:

Voice isolation seemed to be a problem as I airplaying my laptop to my TV to make it group experience. It did start to hear it's own voice, could be counteracted with lowering the volume.

Sometimes the voice reply would just stop but the text was there.

There was some bug where some characters if you went back to them would just answer the last question again. Also just strange stuff like no one knew what they had for dinner but the doctor could tell me how to treat a heart attack. It seemed like the knowledge base they are drawing from didn't have that. Was interesting to see the limits of what they could say.

Design-wise would be good to have a mute toggle as a keyboard shortcut, as we would then try to discuss what people said in the room and it would pick us up.

Also didn't realise we had to accuse the person they were the murderer, as we didn't know what button to press. That was a confusing design as everything had been action based by the UI at the point but now you are asking the user to perform an action with their voice. It also meant we had to accuse them and then immediately start saying why as we know it might cut us off if there's too long a pause. Maybe that could be a button to say you are ready to accuse then you say who and they would ask why, then you present the evidence step by step.

Would be good to have your own stats if you're logged in. i.e time to complete, etc but I guess that's just on a long list of features to implement.

Overall good experience:

I would like to see more involved mysteries. Liked the atmosphere, the characters. Still not sure about the AI portraits but I think that's because it's obvious it is AI. Just using voice to voice made it really nice to use. TBH didn't even need to look at the screen most of the time.

Realtime voice isn't there yet, as there is still handling interruptions and having to ensure you complete your sentence without stopping. Maybe that could be worked around in the app here but probably should be solved at model level. Basically what Thinking Machines are doing.

Also Sophia was really adamant about staying in the kitchen. I got so annoyed I told her to call it the bouncy castle instead.

londons_explore4 days ago

It was working great... But now it says it can't connect.

Did the server go down or did I hit one of your limits?

MrRowTheBoatop4 days ago

Ran out of OpenAI credits, resolved now

BaselAshraf814 days ago

Consider using a model like supertonic 3 for tts.

Its light weight and can run on web inference.

MrRowTheBoatop4 days ago

okay, I posted my goofy prototype demo from a couple days ago.

anonzzzies4 days ago

It is a nice idea ; I will try it later. Everyone can poop on AI, that’s fine, but this specifically states it is AI and how it is used, so just click on another HN link to cry about AI, not the ones that actually specifically mention it.

dbx8424 days ago

This is really fun, great work!

MrRowTheBoatop3 days ago

Thanks <3

october81404 days ago

AI: Yea I did it.

cindyllm4 days ago

[dead]

aswegs84 days ago

Wow!!! That's awesome. So much AI slop. That here is definitely not. Kept me hooked, even though I am not a crime nut. Much improvements possible for sure, but I can see how this becomes a serious game with a lot of players.

MrRowTheBoatop3 days ago

I’d love to hear your improvements

thrance4 days ago

Meh, it's like the fourth such "interview AI to solve the mystery!" implementation I've seen. It's always so bland, uninspired and boring, and drags on for waaay too long.

fitsumbelay4 days ago

Brilliant

garbagewoman4 days ago

"voice connection failed"

zerratar4 days ago

same here. js error though is 0oxvw2ytjke1_.js:1 POST https://api.openai.com/v1/realtime/calls 429 (Too Many Requests). and auth bearer token in clear js request here towards openai. someone could have just used the token for their own good. cant go directly to openai's apis from the website. huge security risk.

kristophph4 days ago

True, I have a dungeons and dragons combat tracker (https://topoftheround.com) where I let the user generate loot items with different settings for them like rarity or magical/non-magical etc. and I am using a firebase cloud function that talks to the actual AI and returns the result. I also have set a limit to how much that token is allowed to spent per month, hopefully OP has thought of that as well. Interpreting and generating voice probably uses a lot of tokens.

MrRowTheBoatop4 days ago

[dead]

MrRowTheBoatop4 days ago

Sorry! Ran out of OpenAI credits, resolved now

qmmmur4 days ago

Boo. AI Slop. Be critical of what you’re making please.

rajivnaskar4 days ago

[dead]

ZeroDayDreamer4 days ago

[dead]

ksingla0254 days ago

[flagged]

ccforai584 days ago

[flagged]

BorisBinyaminov3 days ago

[dead]

mohammedmsgm4 days ago

[flagged]

runtime_lens4 days ago

[flagged]

wundervault4 days ago

[dead]

just60sec2 days ago

[dead]

wewewedxfgdf4 days ago

I have no problem with AI generated text and story but I just can't stand AI voices. Even a hint it is AI narrated and I check out. I would encourage you to consider getting it voice narrated by real people. I'd rather listen to bad/amateur human narration than AI voices.

As a second point - this is very interesting using voice control as game input are there viable alternatives for realtime voice uasing your own GPU?

9dev4 days ago

The entire game hinges on the premise of being able to narrate and understand dynamic text on demand. Pre-recorded messages don’t work for this.

MrRowTheBoatop4 days ago

I think t's possible to use static voice, but it would create a very rigid experience, you would have to have a ton of generic sounds prepared and identify when to play each one. The fun of this is the fluid conversation. I can imagine the static responses being pretty terrible for unrecognized questions. LLMs provide really good RP.

"Tell me where the knife was hidden?"

"I don't understand your question"

hn-front (c) 2024 voximity
source