Video and live models

3.8WP6: Video try-on model

Base model and design

The offline video model fine-tunes Wan2.1 14B, which is licensed Apache-2.0 and is the base behind most 2026 video try-on papers [97, 80, 71]. It also serves as the teacher for the live student in WP7. The design follows the pattern the field has converged on:

• Garment memory. The garment views from WP3 are encoded once and prepended as reference tokens, with a garment-aware position encoding, so the same garment is looked up in every frame [80].
• No per-frame masks. The model reads the person video directly instead of body masks and pose maps, because those cost more per frame than a distilled generator [85, 84].
• Keyframes from the image model. For training data and for hard garments, FabricVTON's WP3 image model dresses a few keyframes, and the video model generates the motion around them [81, 84].

Video data engine

No public video dataset can be used commercially, so the data is built from scratch. Real, consented footage is always the target; synthetic re-dressing only ever creates the input [102, 85].

Each candidate clip must pass a temporal warping check with RAFT optical flow, frame-to-frame and garment similarity checks, identity and background checks, a vision-language judge, and a human spot check [106]. The annotation tools are chosen for their licences: SAM 2 for segmentation, RTMPose for pose, RAFT for flow and Qwen2.5-VL-7B for captions [107, 108]. OpenPose, Sapiens, CoTracker, SMPL-X and InsightFace are excluded, because their licences forbid commercial use.

Training recipe

3.9WP7: Live try-on

From offline model to live model

The live model is a 1.3-billion-parameter Wan2.1 student, in its VACE or Fun-Control variant, that learns from the 14B offline model, following the path LiveVVT and Vidu S2 describe [71, 86]. Before any training, the team tests JD's Apache-2.0 JoyAI-Video-Edit zero-shot on garment changes and benchmarks the leading commercial engine as the quality bar [95]. The student is then built in four steps:

• Causal conversion. The student learns to generate frame by frame from the teacher's outputs, so it never needs future frames [87].
• Self-forcing and step distillation. It is trained on its own rolled-out frames with a key-value cache, using the Apache-2.0 Self-Forcing code, then distilled to 1 to 4 steps, so errors do not accumulate and each frame is fast [89].
• Persistent garment memory. The garment photo and description are encoded once per session and kept alongside a frontal keyframe, which keeps a print the same print as the shopper turns [71].
• Speed engineering. 8-bit arithmetic, compiled kernels and an upscaler from 512p to 720p come after quality is proven.

Streaming system

Figure 2 shows one frame of the loop. The browser sends the camera over WebRTC to a GPU in the nearest region; the model draws the new frame; it streams back to the screen; and the next frame follows about 33 milliseconds later. FabricVTON's existing live page already handles the camera, the consent screen, the session limit and one-time session tokens. Only two small parts of it talk to the rented engine, so FabricVTON's own model can replace it behind the same page. The streaming servers use the open-source LiveKit WebRTC server, and each stream needs about 1.3 to 2.4 megabits per second in each direction [109, 98]. People notice a delay in a mirror-like view at about 150 milliseconds, which open models cannot reach yet, so the design target is about 0.4 seconds from camera to screen. Live serving uses GPUs with hardware video encoders, such as the L40S or RTX PRO 6000, because the H100 and B200 have none.

3.10Evaluation protocol

The primary outcomes, human-rated garment fidelity and pairwise preference, are fixed before experiments begin. Results carry bootstrap confidence intervals and are broken out by garment family, presentation, photo or video condition, skin-tone group and body shape.

3.11Research questions and hypotheses

3.12Compute estimate