Hacker News

Funny this is almost EXACTLY what I did a few days ago on the same machine using very similar techniques and was on the front-page of HN as well:

https://news.ycombinator.com/item?id=48222733 https://blog.simbastack.com/indexed-a-year-of-video-locally/

I wasn't familiar with your project though, interesting stuff.

I'm trying to add more photography related features to Framedex but yeah there's so much we can do locally, exciting times.

Something I've enjoyed more than I expected is Google and Apple photos sending me photo memories and compilations of various things in my life and my kids lives over the last decade.

I'm really bullish on taking more video of my kids, with the thought that it will become easier and easier for AI to put them into little compilations I can enjoy later.

> Then, run the frame analysis pipeline, which will divide the video into separate video scenes (1s each, or 1fps) > (…) > Frames analyzed 57,537

Aha, it makes total sense. This number sounds much more reasonable than “669 GB”, since the actual total size of processed frames would be like 10-30 GB.

(Not downplaying anything. Doing-at-home always requires some math on practicality)

> Total compute time 67h 40m 42s

I’m just curious tho — is there any paying options that can accelerate this kind of process? Just spin up GPU instances?

DaVinci 21 has indexing built-in (AI IntelliSearch). Not to diminish the work you did, but this is now available to many users (probably only Studio users since it has AI in the name)

Does it work for porn collections too?

I was surprised to learn that the

    M1 Max CPU is an ARM/SoC, comparable to an 11th gen Intel i9

ref: https://www.cpubenchmark.net/compare/4585vs4245/Apple-M1-Max...

the link https://iliashaddad.com/blog/i-indexed-669-gb-of-my-gopro-vi...

it is possible to use apple gpu with containers. either with podman + runkit + recent mesa or with recent vllm-metal from docker https://www.docker.com/blog/docker-model-runner-vllm-metal-m...

Would it be possible to upload them all to YouTube as unlisted and then point Gemini to them? Not sure the limitations wrt unlisted videos. Maybe also could’ve been done with the @google photos operator if uploaded to Google Photos if not a corporate google workspace Gemini instance

What models did you use for the stages? I see Qwen2.5-VL-7B-Instruct mentioned as an advanced option, so I assume maybe Qwen2.5-VL-3B-Instruct by default (which is what I also use for a lot of stuff, it is incredibly good at "clean" OCR, but as you maybe indicate not the best at "describing a scene").

EDITED: I didn't realize Whisper was a local model. I never tried transcription before, so I had always figured it was a pay model by OpenAI. I'll have to check it out (although the runtime listed here is a bit daunting).

For that project I'll say I don't see much degradation in embedding quality at much much worse quality than 720p (all the way down to 240p), which speeds things up considerably. Although I don't really do face or object detection, just scene embeddings. To me any process whereby it would take longer to process the video than watch it is probably a no go in general. Obviously a challenge for local-first analysis.

I will be doing these things with local LLMs

Take a fast, small and powerful LLM running locally to index my personal data like images, videos, documents and enrich them and tag with the enriched metadata.

Want to group by people - Search tagged metadata and group it What to search an image by description - tagged metadata What to organize by anything - tagged metadata

This should (hopefully) put an end to my file clutter

I have an RTX 5090 card but it only has 32 GB RAM, can something like this work on my machine?

Thanks for sharing! I make videos and often have the same problem as well.

Being able to semantic search over your library is useful, but does it solve the review problem? I feel like you would still need to watch the footage back before you know what you're working with.

Related article on indexing videos but with a local text description and using Gemma4: https://blog.simbastack.com/indexed-a-year-of-video-locally/

Well done! I couldn't understand how you are building reels out of it via the agent. Is it some sort of AI tool calling that takes image links and builds a reel via some video editing tool ? Or +/- time delta around the timestamp returned from the indexed from a given query + join them together?

if anyone is interested in searching large video collections local and offline I suggest taking a look at Jumper https://docs.getjumper.io

comes with some nifty features like NLE- integrations, people search, MCP, API etc

Disclaimer: one of the co-founders

Cool build but the example videos you provide at the end are . . . not what I would hope for when thinking about the highlights of 2000+ videos of biking? For example the dog barking video only has one scene repeated two or three times and it's five seconds long?

I've been screenshotting twitter since 2016 when Brexit happened and with the goal of one day putting together some kind of art piece.

The world and our discourse around it has changed so much over the past ten years and now with this kind of technology I'm so excited to be able to classify these images from my iCloud and start on the project.

I’d like to see embedding of actual video clips become practical in this type of workflow.

Frame level embedding it covering a lot, but can miss out on a lot of action related searches.

Thank you so much for your support over Hacker News, I wanna share with people of HN. a special discount code "HACKERNEWS", use it here https://shop.edit-mind.com/checkout/buy/9f18a6f0-b437-47ec-b... and get 10% OFF only for the first 5 people and it'll expires

This would fit most best as a “Show HN:” post :)

I wonder how long it would take on faster hardware. I have ten times that much footage, but 67 * 10 hours is a lot of processing.

I might be better off getting something with a beefy GPU on AWS or Google cloud.

https://archive.is/O6CLQ

When trying to read this article, the main website was throwing errors to CloudFlare unfortunately

https://iliashaddad.com/blog/i-indexed-669-gb-of-my-gopro-vi...

> Many of the videos I captured amazing moments, and sometimes it's kind of hard to watch the full videos to get those moments.

Yep. I had the same problem.

> Then, run the frame analysis pipeline [...] I have a face recognition plugin using my custom faces data, object detection, on-screen text, shot type, and scene description [...] we will have three vector DB collections that have all the information about our videos, like video location metadata, camera name, faces recognized, objects detected, on-screen text, transcription, description of each scene, and many more [...] we can get better indexed data if you use the advanced mode indexing to use the Qwen2.5-VL-7B-Instruct model to understand and describe your video much better, but at a slower indexing speed

Yeah, uhm... ok :)

If anyone else has a similar problem, the real solution is as follows:

1. When recording, if you witness an interesting moment worth saving later, press the power button — this will mark the current moment in the video as a chapter.

2. Find the chapters later when editing and cut them into clips.

3. You're done :)

This has two main benefits over the insanity above:

1. It's trivially simple instead of insanely complex and inefficient.

2. It will reliably catch all the stuff you find interesting, since you're the one doing the marking.

The downsides:

1. Doesn't work retroactively.

2. It may miss interesting stuff if you miss it at the time as well.

3. Only works for this use case.

4. Nerds won't salivate over your usage of cutting edge tech.

I would love your feedback and suggestions for new improvements or features you wanna have, either in the source available version, the desktop app or blog post itself?

this is really cool. was looking to do something similar on mbp 64gb

Grab frames, lower res, classify, combine meta data. Write to sql

can vlm be used instead or it's too heavy and slow

Now this ^^ is an awesome use case!

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

[dead]

[flagged]

A lawyer I know who specialises in rape, and is excellent at getting the obviously guilty exonerated, lost a case last year because of GoPro videos.

Her client was recording while committing the abhorrent crime. The criminal would otherwise have got off.

From my perspective, the GoPro camera produced a good outcome. Still, one has wonder why anyone to record their criminal actions.