The person stays on the sofa, sitting the same way, moving the same way. Only what's behind them changes: the plain wall is gone, and a rooftop at dusk is there instead. The camera move never shifts and nothing about the person is redrawn. That's the clip above, and it's what AI background replacement video should look like: one thing changes, everything else is proof it didn't have to.
Most searches for this land on a quick photo-style background remover that leaves a flickery edge the moment the subject turns their head, or on the assumption that a clean swap needs a green screen on set. Neither is true anymore, but getting a stable result takes more than one click.
Why a photo background remover isn't enough
A still-photo background changer only has to cut one convincing edge, once. Video has to keep that edge right as the subject moves, and that's where the quick tools fall apart:
- The mask has to track, not just cut. A single segmentation pass on frame one doesn't follow a head turn or a raised arm.
- The new background has to stay put relative to the camera. If the camera pans and the background doesn't move with it, the swap reads as a sticker instead of a place.
- Edges that break the mask (hair, motion blur, a hand crossing in front of the subject) are exactly where a one-pass cutout shows its seams.
Chroma key solves the "pull a clean matte" problem by controlling the color on set. A tracked mask solves the same problem without needing a green screen in the first place, which is why it works on footage you already have.
The workflow that holds up
1. Mask the subject, not the background
Draw the mask around the person or object that's staying, on the clearest frame you've got. It's easier to track a clean subject outline than to chase a changing background around them.
2. Track it through the whole shot
Let the mask follow the subject as they move, then scrub through at full size. The frames to check are head turns, a hand passing in front of the body, and anywhere the subject partly leaves frame.
3. Generate the new background to match the camera move
The replacement has to move the way the original background would have: if the camera pushes in or pans, the new background needs the same motion, not a static plate sitting behind the subject.
4. Match the light
A rooftop at dusk behind someone lit for daylight indoors is the giveaway. Check that the direction and color of light on the subject still makes sense against the new background before you call the shot done.
5. Check the edge at 100%, not a shrunk preview
Halos and soft fringes around hair or fast motion only show up at full size. If a handful of frames are off, fix the mask on just those frames and rerun that range.
Asking an agent to do it
In Shoty, you can run the steps above by hand or hand the instruction to the agent in the chat:
Replace the background behind the person on the sofa with a rooftop at dusk. Track them through the whole shot, match the camera's pan, and keep the person and their motion exactly as they are. Show me the result before you replace the original.
With the run mode set to review, the agent masks the subject, tracks it, generates the new background and waits for your approval before anything overwrites the original clip.
Where it still needs you
- Hair and fast motion at the mask edge are still worth a manual check, even after tracking gets most of the way there.
- Light matching is a judgment call. The tool can get close; deciding if it's convincing enough to ship is yours.
- Big background changes (a totally different time of day, a crowd behind the subject) mean more of the frame is invented, so review those harder than a simple wall swap.
Download the free Shoty editor to try a background swap on your own footage. If the subject also needs isolating frame by frame first, see AI rotoscoping, or for the full list of what AI VFX covers, AI VFX: remove, replace, extend, relight or composite a shot.