Ask an agent to "swap the background behind the sofa for a rooftop at dusk" and in the clip above, it does: the people, their motion and the camera move stay put while only the background changes. The part that makes that kind of MCP video editing possible isn't the model, it's the tool bridge between the agent and the video project, and that bridge is MCP.
This guide is about that bridge: what it is, which editing tools on it actually matter, and how it's wired up so any agent CLI can use the same ones.
What MCP is, without the protocol spec
Model Context Protocol is a standard for exposing a set of callable tools to an AI model, the same way a plugin exposes functions to an app. An agent connected to an MCP server can call a tool, get a structured result back, and decide what to do next, instead of only producing text.
For video, that distinction is the whole difference between an agent that talks about editing and one that edits:
- Without a tool bridge, the agent can only describe steps. You read them and do the work.
- With an MCP server exposing the right tools, the agent calls "look at frame," "move this clip," "generate this shot," and the project itself changes.
The model doesn't need new training to do this. It needs a server standing between it and the project, offering tools worth calling.
The tools that actually matter
Not every tool on an MCP server moves the needle. For editing, four kinds do:
- Vision. A tool that returns a frame or a contact sheet, so the agent checks what's in a shot before cutting or replacing it, instead of guessing from a filename.
- Timeline read/write. Tools to place, trim, move, layer and re-time clips in the actual project, not a text description of a project.
- Generation. A tool that hands a shot to a video or image model with references and returns a result the agent can place.
- Export. A tool that hands the finished cut to another app in a format it can open.
A server that only offers one or two of these leaves the agent doing homework again for the rest. All four together is what makes "cut the pauses, swap the background, export it" a single instruction instead of three separate tools you operate by hand.
How Shoty wires this up
Shoty is a desktop video editor (Mac and Windows) with an agent chat built in. Pick an agent in the model picker and Shoty runs that agent's own CLI, signed in with your own account, Claude Code, Codex, or Gemini's CLI. Shoty never sees the provider token and charges no fee for the agent itself.
For each turn, Shoty opens a localhost MCP server and exposes its editing tools through it: look at an image or frame, read and update the timeline, generate a shot, update project memory, export. Because it's the same server and the same tool set regardless of which CLI connected, the agent's capability doesn't change when you switch models, only its judgment does.
The timeline underneath is a text format (.shoty), so the agent is reading and editing a structure it can reason about instead of inferring one from pixels. Every change is undoable, and you set how far the agent goes on its own: suggest, preview, or auto, with run modes from Auto-review up to Run Everything.
Ask the agent, not the terminal
Once an agent is connected over MCP, you don't configure tools, you just ask for the edit:
Look at shot 3, replace the background behind the sofa with a rooftop at dusk, and keep the two people, their motion and the camera move exactly as they are.
Export the current cut as an EDL so I can finish the grade in DaVinci Resolve.
Each of those is one call to a different tool on the same server: vision, generation plus timeline write, and export. The agent chains them without you naming which tool to use.
What the bridge doesn't do for you
- It doesn't pick the right take. The agent can generate and place several versions of a shot; deciding which one is right is still yours.
- It doesn't replace review. Generated shots use credits, so set the run mode to review if you want to approve them before anything renders.
- It doesn't make the agent smarter. MCP gives any connected model the same tools. A harder edit still benefits from clear instructions, naming what has to stay the same as much as what should change.
Download the free Shoty editor to try it with Claude Code, Codex or Gemini. For a walkthrough with one specific agent, see Claude video editing.