Analyzing video clips with AI: what works today

Most writing about analyzing video with AI is either a demo reel or a warning. Here is the useful version: signal by signal, what a model gets right on real footage, what needs a person looking at it, and what it still fails at in ways waiting will not fix.

One thing I will not do is quote accuracy percentages. I have no benchmark on your footage that I would defend in front of you, and neither does anyone else selling you this. Treat any vendor number as a claim about their test set, not your archive.

Speech: the most reliable signal you have

Transcription works well enough to trust as a search index on the first pass. Clean dialogue, a decent mic, one person talking, and you get text you can search across a whole shoot. For interview work this alone justifies the pass, because it turns hours of footage into a document you can read, which is how documentaries were always written. That workflow is the paper edit.

It runs on your Mac. ClipLogger uses Apple's on-device frameworks by default and offers optional Whisper models: a base English model of about 150 MB, and Whisper Large v3 Turbo at about 1.6 GB, which is multilingual and noticeably more accurate. Nothing leaves the machine for either.

The failures are predictable. Two people talking over each other degrades badly, though per-channel transcription helps if you recorded to separate channels. Wind, crowd noise, and a camera-top mic across a room all cost you. And the one that catches people out: proper nouns. Names, streets, and brands come back plausible and wrong, not obviously garbled. A transcript is a fine search index and a poor quotation source.

On-screen text: excellent when it is legible, absent when it is not

Optical character recognition reads text out of the picture: slates, scoreboards, lower thirds, street signs. On a shoot that uses slates this is close to free structured metadata, and it is why a scoreboard can anchor a whole game day.

It fails on motion blur, glare, low contrast, stylized or handwritten type, and text that is small in frame. Sampling helps: ClipLogger picks frames by sharpness rather than at fixed intervals, so it reads the crisp frames instead of the pans. But there is no partial credit. Either the characters were legible in some frame or the value is not there, and an absent value is honest in a way a guessed one is not.

Faces: grouping, which is not recognition

This distinction gets blurred constantly in marketing. What runs on your footage is face grouping: similar faces cluster together, so you name a group once and find that person everywhere. It ships no celebrity database and identifies nobody on its own. A group has a name because you gave it one, or your imported roster did. All of it runs on-device.

The failure modes are the ones you would guess and one you would not. Backs of heads, hard profiles, hats and sunglasses, and low light produce no grouping at all, which is fine. The subtler problem is that one person can fracture into two groups (different lighting, four years apart, a beard) or two similar-looking people can land in one. Both are why groups are presented as evidence rather than written as fact.

A person is not a face, though. A person is an identity carrying several kinds of evidence at once: a face group, an appearance, a jersey number bound to a roster. When those signals disagree, the right behavior is to show you the conflict and take your ruling, not quietly pick one. That model is subjects: one identity, many signals.

Framing, motion, and quality: cheap triage

Framing and motion classification and an on-device aesthetics score are the least glamorous outputs and among the most useful, because they are good enough for sorting and nothing depends on them being exactly right. Sort a 400-clip day by quality to surface the best frames, filter the cull candidates to sweep the junk the other way, and you have cut the pile before making a single real decision. Use them as a lens, never a verdict: the score does not know that the slightly soft shot is the only take where the line landed. Your flag is the decision, the score only changes the order you see things in.

Clip reasoning: the narrative, and its real limit

This is the part people mean when they say AI video analysis: a model that looks at the picture and tells you what is happening. It needs a vision model. ClipLogger can run one locally after a one-time download (the default is Qwen2.5-VL 7B), through your own API key, or through the Rush cloud lane.

Here is the limitation that resets expectations correctly, and almost nobody states it: the model does not watch your clip. It reads a handful of sampled frames, six by default and adjustable from two to sixty, at 768 pixels on the longest edge, alongside the transcript, the on-screen text, and your field definitions. Long clips scale the sample up automatically. But a thing that happens between two sampled frames did not happen as far as the model is concerned. For a static interview that is irrelevant. For an eight-second play, it is the whole game, and it is why you turn the frame count up when the action is brief.

The other structural limit: the model answers in your vocabulary or in generic prose. Hand it a closed field with a one-line instruction and it must answer inside your options. Hand it nothing and you get competent, useless description. That is the argument for defining fields first, in a metadata schema for logging video.

What still fails, and will keep failing

Intent. Which of six near-identical takes is the one, because that judgment was never in the frame. Off-screen knowledge: that the woman in the blue coat is the client's mother, that this building is the one in dispute. Continuity across clips. Reliable counting. Any question whose answer depends on what the project is for.

None of this is a complaint about model quality. These are questions the footage does not contain the answer to, and no amount of scale puts information into frames that was never recorded.

Local versus cloud, briefly

Local is free, private by construction, and capped by what fits on your Mac. Cloud runs larger models that read hard frames better: the number half-hidden by a shoulder pad, the scoreboard behind glare. The difference is real, and I say so even though we sell the cloud lane. It is a per-batch routing call, not a belief system, argued in on-device vs cloud AI is routing, not religion; what must never happen either way is frames leaving silently, which is what privacy should mean for footage tools.

Everything above lands as a suggestion, for the reason in AI proposes, you confirm. The category is what an AI video logger actually does, the fields these signals answer into are a metadata schema for logging video, and running the passes against shared storage is an AI assistant for your NAS.

Analysis is good enough today to do all the watching and not one bit of the deciding.