GenRuntime
A local video studio for a single 16 GB GPU. You give it a JSON file describing the shot, and it gives you back a clip that has passed quality checks, with a person approving the keyframes before any GPU time goes into video.
What you see
A real shot from start to finish, with a swordswoman walking through a bamboo forest. The graph shows the spec, the keyframes, the review, the video with its QA and the clip ready to download.
AI-generated clips
Press play: they all have sound.
00:00 / 00:36
The silence of the automatons
A short film in a wheat field, where more and more machines keep appearing.
00:00 / 00:14
Platform in the fog
A red train arrives through the mist and passes someone waiting with their back to us.
00:00 / 00:05
Bamboo swordswoman
The shot in the screenshots below: she walks toward the camera and says her line.
00:00 / 00:06
Lofi cat
Rain on the window, a steaming mug and a cat watching the night go by.
00:00 / 00:05
Clay ghost
A ghost peeks out from behind the gravestone, with a clay texture.
00:00 / 00:07
In the rain
A yellow raincoat, a wet street and a glance at the watch.
How it flows
- JSON spec
- Linter
- FLUX.2 keyframes
- Human approval
- LTX-2.5 video
- QA and final clip
Architecture
GenRuntime is not one program. It is two containers with dependencies that cannot share an environment, a host that coordinates them, and state on disk that lets every shot resume where it stopped. On a single 16 GB GPU every run counts, so the system avoids repeating work it does not need to repeat.
Spec and linterPython
Each shot is a JSON file. The linter blocks failures we already know about, such as the VRAM budget, dialogue timing, hands, fades and quadruped jumps, before the GPU is touched.
KeyframesFLUX.2 klein
Generates candidates per keyframe and filters them by character count, design invariants, location with OWLv2 and camera from geometry. The candidate that changes the background least wins.
Human reviewHost
The pipeline stops at review.png until someone approves. Approval is tied to the spec hash, so it does not carry over to another version.
Video and audioLTX-2.5 DFR
Renders the take with Lightricks' stack, in its own container. Each run leaves a record with the command, the timings and the real VRAM peak.
Quality controlVLM, DINOv2, Whisper
Checks content, camera, likeness to the keyframes and dialogue. A take is delivered only if it passes every check. If it fails, it is retried with another seed.
- DockerTwo images with isolated dependencies
- OllamaVision model for questions about a frame
- OWLv2 and DINOv2Locating characters and measuring similarity
- WhisperTranscribes the dialogue to verify it
- React FlowThe interface graph is the same spec the pipeline consumes
Several processes, not one
The two containers pin incompatible versions of torch and the models. The host coordinates and the containers do the GPU work, so no dependency fights another.
State lives on disk
output/studio/<name>/ keeps the progress. Running it again continues where it stopped, and if the spec changes only what depended on it is cleared.
Rules born from real failures
Every linter rule and every QA threshold answers a failure that actually happened, not a hunch. That is why the thresholds are concrete, such as a 0.80 similarity to the keyframes.
Measure before judging
Every QA verdict rests on a measurement, such as the background homography with the subjects masked, the DINOv2 similarity or the Whisper transcript. The VLM only answers simple questions about one frame.
Retry with judgment
A QA failure is not always bad luck. The experimental runner classifies each failure, and it rolls the seed again only when the failure looks random, and aborts when the problem is in the spec.
How it is built
System map
Three environments, each with its own stack. The host decides what to do and the containers do the GPU work.
Host
- studio: the pipeline core
- lab: experimental runner
- Graph interface
- State on disk
docker run · keyframes and QA
App container
- FLUX.2: keyframes
- Video QA with OWLv2 and DINOv2
- Tests that run without GPU
docker run · video takes
ltx25-dfr container
- LTX-2.5 DFR
- Lightricks code, unmodified
- OllamaVision model (VLM) for questions about a frame
- WhisperTranscribes the dialogue to verify it
- GPUOne card with 16 GB, one GPU job at a time
One shot, end to end
Every stage leaves an output you can review. The approval gate is the only one that waits for a person.
- JSON specDescribes characters, set, action, camera and dialogue.
- LinterBlocks known failures before the GPU is used.
- KeyframesFLUX.2 generates candidates and filters them by character count, invariants, location and camera.
- Human gateStops at review.png until someone approves. Approval depends on the spec hash.
- Video takeLTX-2.5 DFR renders video and audio in its own container.
- QAChecks content, camera, likeness to the keyframes (DINOv2) and the dialogue (Whisper).
- Deliveryoutput/<name>.mp4 and its report, only if the take passed every check.
If QA fails, it retries with another seed (+1000), up to three attempts by default.
What to do with each failure
The experimental runner does not retry everything the same way. Each kind of failure has its own action.
PASSDelivers the clip.QA_SUSPECTReviews the same take across three neighboring frames before deciding.STOCHASTICThe failure looks random: tries another seed.INFRAIt comes from the environment, not the shot: retries the same seed once.STRUCTURALThe problem is in the spec: aborts so it can be fixed.
What stays on disk
Progress lives in folders, not in memory. Running it again picks up where it stopped, and if the spec changes only what depended on it is cleared.
output/studio/<name>/
- spec.jsonThe current version of the shot
- keyframesCandidates, with the chosen one marked
- review.pngWhat the person checks before the video
- APPROVEDHash of the approved spec; it is not valid for any other version
- takesEach attempt with its seed and its record
output/<name>.mp4 + _report.jsonOnly appears if the take passed the full QA.
The challenge
Generating video on a single 16 GB GPU leaves little room, and a bad shot tends to show up late, after the render has already used its time. The challenge was getting from a description to a reliable clip on my own machine, with checkpoints before GPU time goes into video, and with a system that can explain why a shot failed.
What's behind it
- Each shot starts as a JSON file that passes through a linter before the GPU is touched.
- FLUX.2 generates keyframe candidates and filters them with computer vision. Characters are kept as references so they stay the same across shots.
- A person approves the keyframes. Only after that is the video rendered.
- LTX-2.5 generates video and audio inside its own container, without changing its code.
- Quality control combines specialized models. Each one measures a single thing, and the VLM only answers simple questions about one frame.
- A graph interface shows every stage. The graph is the spec, so what is drawn compiles to the same JSON the pipeline consumes.
- The experimental multi-shot runner predicts VRAM and time, and classifies failures to decide whether to retry or abort.
Decisions and trade-offs
- A person approves before video. It slows the flow, but it avoids spending a video run on keyframes that already look wrong.
- Two Docker images instead of one. There is more to maintain, but the dependencies of each part conflict, so this way each keeps its own.
- Fully local instead of a cloud API. The 16 GB limit shapes every model choice, and in exchange I do not depend on a provider.
- The multi-shot runner is experimental. It saves GPU time in batches, but it is not the main flow yet, so I do not present it as one.