The clip above is a studio shot where one performer has been swapped for another. Nothing else moved: the same light rig, the same camera push, the same pose carried through. That's the bar for "replace actor in video ai" to actually work: everything except the person stays untouched.
Most results that go wrong aren't wrong because the new face looks bad on one frame. They're wrong because the swap doesn't track the original performance, so the replacement slides, flickers, or sits under the wrong light as the shot moves.
Before anything else: whose footage, whose likeness
Swapping a performer means generating a real person's likeness over someone else's performance. Only do this with footage you have the rights to, and get consent from anyone whose likeness is going in or out of the shot. That's not a technical step and this guide won't tell you what's allowed where you are; it's a question to answer before you track a single frame.
Why a straight face swap looks fake
A photo swap only has to match one still. A video swap has to match:
- The performance, frame by frame: head turns, blinks, the exact timing of a gesture.
- The camera move, so the replacement doesn't drift relative to the background as the shot pans or pushes in.
- The lighting, including shadows the original performer cast and any light hitting them from off-screen.
Miss any one of those and the eye catches it even if the new face looks perfectly rendered in a single frame.
The workflow that works
1. Track the performer, not just their face
Mask the whole performer and track it across the shot, including frames where they turn away from camera or are partly blocked. The replacement is generated against this tracked region, so it follows the same motion instead of being pasted frame by frame.
2. Carry the lighting with it
Note where the light is coming from in the original shot. The replacement needs to pick up the same key light and cast a shadow in the same direction, or it reads as a different shot stitched into this one.
3. Generate the replacement on the tracked region
Run the swap on the tracked mask, not the whole frame. Keeping everything outside the mask untouched is what keeps the background, other people in the scene and the camera move exactly as they were.
4. Check the edges and the handoffs
Play the result at full size around the mask edge, and on any frame where the performer overlaps something (a hand on a prop, a shoulder against a wall). Those handoff frames are where a swap most often slips.
Asking an agent to do it
In Shoty you can run each step by hand, or ask the agent in the chat:
Replace the performer in shot 3 with the reference I'm attaching. Track their full performance through the shot, match the key light and shadow direction, and keep the camera move and everyone else in frame untouched. Show me the result before replacing the original.
With the run mode set to review, the agent tracks the performer, generates the replacement and waits for your approval before anything overwrites the original shot.
Where it still needs a human
A swap that nails pose and lighting can still carry the wrong expression at a key emotional beat; that's a performance call, not a tracking error, and it's yours to make. And the rights question from the top of this guide doesn't go away once the shot looks right: that decision stays with you, not the tool.
Want to try it on your own footage? Download the free Shoty editor. For removing something from a shot instead of replacing it, see how to remove an object from video with AI.