Fireworks burst over a rooftop that was plain sky a moment earlier. The people in frame, the camera move and the rooftop itself don't change; only the sky above them does. That's an AI VFX composite: a new element added into a real plate, matched to the shot's light.
"AI VFX" isn't one feature. It's six different jobs that all start the same way: track a mask across the shot, then generate, remove or relight inside it.
The six jobs, and what's actually hard about each
- Remove: take something out and rebuild the background behind it, including any shadow or reflection it left.
- Replace: swap a background or a performer while the rest of the shot (camera move, lighting, everyone else) stays put.
- Extend: outpaint the frame, for a new aspect ratio or to widen a shot that was framed too tight.
- Track: follow a subject or object through the shot so a mask, a title or an effect moves with it instead of sliding.
- Relight: change the time of day, weather or light direction after the shot was filmed.
- Composite: add a new element (fire, smoke, fireworks, a creature) into real footage so it reads as if it was there.
What they share is the failure mode. A photo tool only has to produce one convincing frame. Video has to hold that result steady for hundreds of frames while the camera and subject move, and it has to get the edges right: a shadow that didn't move with its object, a reflection that got left behind, a halo where a mask was drawn too tight. Any one of those breaks the illusion even if every frame looks fine in isolation.
A workflow that holds up across all six
1. Mask and track first, generate second
Draw the mask once and let tracking carry it across the shot, whether you're removing an object or isolating a performer to relight. Scrub through and fix the mask only where it drifts: turns, partial occlusion, motion blur.
2. Name what must not change
"Change one thing, keep the rest." Say explicitly what's locked: the character's face and performance, the camera move, everyone else in frame. That constraint is what keeps a generative pass from repainting the whole shot instead of the part you asked for.
3. Match the light before you composite
For relighting or adding an element, check where the existing light is coming from and match it, including the shadow and any reflection the new element should cast. A fireworks burst with no corresponding light on the people below it is the giveaway that it was dropped in rather than shot there.
4. Review at full size, frame by frame near the edit
Play the result at 100% around the mask edge and across the frames where the subject or camera moved fastest. That's where halos, shimmer and drift show up first. Fix the mask on just those frames and rerun that range instead of the whole shot.
Asking an agent to run one of these
Add fireworks over the rooftop in shot 4, matched to how the current light hits the people in frame. Keep the people, the camera move and the rooftop exactly as they are.
With the run mode set to review, the agent generates the shot and waits for your approval before anything replaces the original. You still decide whether the light match is convincing enough to ship.
Where it still needs you
- Matching light and shadow on a composite is a judgment call a tool can get close on but not guarantee; check it at full size before you commit.
- Big holes on removal or extension mean more of the frame has to be invented, so check those shots harder than a small cleanup.
- Consent and clearance on replacing a real performer or a recognizable logo is a decision for you, not the tool.
Most AI VFX work on a timeline is one of these six jobs applied with a tracked mask and a clear instruction about what stays locked. Download the free Shoty editor to try removal, relighting or compositing on your own footage. For the object-removal case specifically, see how to remove an object from video with AI, or for isolating a moving subject, AI rotoscoping.