The clip above swaps the person walking down a hallway for someone else. The walk, the camera move and the hallway light all stay. Every shot like that starts with the same unglamorous step: a clean mask of the person on every frame.
That step is rotoscoping. Done by hand, it's the most hated job in post. Here's how AI changes it, and where it doesn't.
What AI actually does in rotoscoping
Traditional roto means drawing splines around the subject on keyframes and fixing the in-betweens. A two-second shot can be a day of work.
AI rotoscoping splits the job in two:
- Segmentation. You click the subject or draw a box on one frame. The model outlines it.
- Tracking. The mask follows the subject through the shot, adapting as it turns, gets occluded and changes shape.
Your job shifts from drawing every frame to reviewing every frame and fixing the few that broke.
Where AI roto still breaks
Be honest about these before you promise a client a turnaround:
- Hair and fur. Fine strands against a busy background. Expect to check edges.
- Motion blur. A fast arm is semi-transparent for a few frames. A hard mask edge looks cut out.
- Overlaps. Two people crossing, a hand passing in front of the face. The track can jump to the wrong subject.
- Leaving and re-entering frame. Re-anchor the mask when the subject comes back.
None of these are reasons to roto by hand. They're the frames you review first.
A workflow that holds up
1. Mask on the clearest frame
Start where the subject is sharp and fully visible, not on frame one. Tracking forward and backward from a clean frame beats tracking from a blurry one.
2. Track, then scrub at full size
Scrub the whole shot at 100% and mark the frames where the edge breaks. Usually it's a handful, clustered around the problem spots above.
3. Fix per frame, not per shot
Correct the mask only on the broken frames and rerun just that range. Throwing the whole track away because three frames failed wastes the AI's best work.
4. Use the mask for the real job
The mask is never the deliverable. It's what lets you replace the person, relight them, replace the background behind them or composite effects between them and the set. In Shoty the mask stays attached to the shot, so the next step reuses it instead of starting over.
Asking the agent
Mask the man in the hallway shot through the whole clip. Check the frames where he passes the door and fix the mask if his arm gets cut off. Then replace him with the character from the reference sheet, keeping the walk and the lighting.
The agent does the masking, checks the frames you named and runs the replacement. You review the edges where you know it's hard.
Download the free Shoty editor to try it on your own footage. Related: how to remove an object from video with AI.