SeedVR2 3B video restoration¶

The native SeedVR2Pipeline restores an input video using the released 3B FP16 NaDiT and s8/c16/t4 causal VAE. It uses fixed checkpoint conditioning, CFG=1, and one Euler step. Text prompts do not change conditioning.

Code ownership¶

The model package groups each computation stage in one file: nadit.py owns window geometry, routing, RoPE and the DiT; vae.py owns the causal VAE and its temporal/spatial tiling; pipeline_seedvr2.py owns input validation, decoding, conditioning, the Euler step and colour transfer. config.py owns model limits, and long_video.py owns the optional HTTP job/window lifecycle. Framework code only registers the model, its pre/postprocessors, metadata and HTTP routes.

color_correction_method is model-specific: online requests use the existing extra_params JSON object, and offline sampling uses extra_args={"color_correction_method": "wavelet"}. The default remains lab; invalid values are rejected by model-side validation before inference. It is not a top-level field in the common video API.

Model directory¶

Place these files together, retaining their names:

Add model_index.json containing:

{"_class_name": "SeedVR2Pipeline"}

The loader checks every checkpoint tensor, including RoPE buffers, and rejects missing, duplicate, unexpected, or incompatible tensors. Install PyAV for video decoding. The validated environment uses PyAV 18.1, PyTorch 2.13/CUDA 13, and L20 GPUs.

Serve and restore a video¶

vllm serve "$MODEL_DIR" --omni \
  --model-class-name SeedVR2Pipeline --dtype float16 --enforce-eager \
  --num-gpus 1 --host 127.0.0.1 --port 8098

curl --fail-with-body http://127.0.0.1:8098/v1/videos/sync \
  -F 'prompt= ' \
  -F 'input_references=@input.mp4;type=video/mp4' \
  -F 'size=224x128' -F 'num_inference_steps=1' \
  -F 'guidance_scale=1' -F 'seed=7723' \
  --output restored.mp4

For two-rank window sequence parallelism, expose two GPUs and replace --num-gpus 1 with:

--num-gpus 2 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":2}}'

Use degree 4 and four visible GPUs for SP4. Window-SP alone replicates the VAE. To shard VAE activations across the same ranks, use:

--vae-use-tiling --num-gpus 2 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":2,"vae_patch_parallel_size":2,"vae_parallel_mode":"spatial_shard_height"}}'

The VAE patch degree must match the window-SP degree. For a 16-frame 1280×720 input restored at its original size, use four GPUs, VAE tiling, and height sharding with both degrees set to four:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve "$MODEL_DIR" --omni \
  --model-class-name SeedVR2Pipeline --dtype float16 --enforce-eager \
  --vae-use-tiling --num-gpus 4 --distributed-executor-backend mp \
  --stage-overrides '{"0":{"ulysses_degree":4,"vae_patch_parallel_size":4,"vae_parallel_mode":"spatial_shard_height"}}' \
  --host 127.0.0.1 --port 8098

Request size=1280x720 for this landscape input or size=720x1280 for a portrait input of the same dimensions. Keep the requested output size equal to the input size when testing restoration without upscaling.

The multipart API requires a prompt field; a single space supplies a blank prompt. Nonblank text is rejected. Omit fps to retain the source frame rate; an explicit rate must match the source. Output dimensions are explicit multiples of 16. Input frames are resized with antialiased bicubic interpolation. The final frame is repeated internally to reach 4n+1, and the decoded output is cropped back to the original frame count. Five frames remain five; six frames are internally padded to nine and return six.

Colour correction¶

Restoration reproduces detail faithfully but shifts global colour, so the resized input is used to transfer colour back onto the restored frames. This is on by default; pass extra_params={"color_correction_method":"wavelet"} to choose how:

curl --fail-with-body http://127.0.0.1:8098/v1/videos/sync \
  -F 'prompt= ' -F 'input_references=@input.mp4;type=video/mp4' \
  -F 'size=224x128' -F 'num_inference_steps=1' -F 'guidance_scale=1' \
  -F 'seed=7723' -F 'extra_params={"color_correction_method":"wavelet"}' \
  --output restored.mp4

lab and wavelet retain the restored high-frequency detail; adain measurably softens it because it rescales every frequency. The transfer runs in FP32 after the VAE decode and costs a fraction of a second per frame. Unrecognized values are rejected with 400 before the clip is admitted. Correction cannot recover colour lost to 4:2:0 chroma subsampling in the returned MP4, which dominates the remaining deviation once the transfer is applied.

Request admission¶

The default 3B limit is 848×480 pixels per input or output frame and 2,035,200 pixels across the requested output clip. With VAE tiling and window SP of at least four ranks whose vae_patch_parallel_size matches ulysses_degree, the per-frame limit rises to 2560×1472 and the clip budget to 18,841,600 pixels. Raising the degree never lowers the budget. Both budgets count temporal padding to 4n+1 frames: the default admits up to five 848×480 frames or 93 192×112 frames; the sharded profile admits up to 45 848×480 frames or five 2560×1472 frames. A 16-frame 1280×720 or 720×1280 clip pads to 17 frames and uses 15,667,200 of the sharded clip's 18,841,600 pixels. Both orientations passed original-size HTTP restoration on four RTX 5090 GPUs with VAE tiling and height sharding. An independent 257-frame cap bounds per-frame decoder work for tiny inputs. The other longer combinations above are admission bounds, not completed GPU validation. The decoder checks declared duration, frame count, and input dimensions when available, then enforces the limits as frames arrive. Requests outside these budgets return 400 before building the resized whole-clip tensor.

The per-frame cap is calibrated for the smallest qualified device. The clip cap scales with the smallest visible device's memory: it stays at the calibrated five 2560×1472 frames on a 32 GB device and grows on larger ones from a per-rank memory model measured at SP4 (see sharded_budget in vllm_omni/diffusion/models/seedvr2/pipeline_seedvr2.py), which keeps 15% of the device free and leaves room for allocator fragmentation. It stops at 2,118,057,984 padded pixels (513 frames at 1536×2688), the largest clip validated on four B300 ranks; 80 GB devices get about 1.08 billion and 141 GB or larger devices reach the cap. Clips at the cap in both memory regimes (561 frames at 2560×1472 and 2,049 frames at 768×1344) peaked at 54 GiB of activations per rank. Set these before starting the server to override the caps; each must be a positive integer, and validating an override on the target hardware is the operator's responsibility:

The sharded profile applies to any ulysses_degree of four or more, so eight ranks inherit a budget calibrated on four. Raising these caps trades a 400 for a CUDA OOM: the admission check passes and the request fails later inside the DiT forward or the VAE decode, which can abort every in-flight request on the engine rather than only the oversized one. Before serving with raised caps, restore the largest clip they admit once and confirm it completes.

A 362-frame 1536×2688 2x upscale was restored on eight ranks with ulysses_degree=8, VAE tiling and height sharding, using 4,300,000 for the per-frame cap, the memory-scaled clip cap and 2,000 for the frame cap. The long-video route sizes its windows from these caps, so the clip cap sets how many frames each window covers.

Long-video restoration¶

POST /v1/seedvr2/restore-long accepts one uploaded 24 FPS video, a blank prompt, size, num_frames (up to 7,200), and optional loop_input=true and extra_params={"color_correction_method":"wavelet"}, which it applies to every window. An output frame must fit the sharded per-frame pixel cap. It returns a job ID; poll GET /v1/seedvr2/restore-long/{id} and download the completed MP4 from GET /v1/seedvr2/restore-long/{id}/content. DELETE /v1/seedvr2/restore-long/{id} asks a running job to stop; it settles as cancelled at the next window boundary. A job whose owning process is gone, such as after a server restart, reports failed on the next poll.

Jobs live on the API server's local disk, so the route requires --api-server-count 1 and runs one job at a time. Storage is bounded by an upload cap and by deleting settled jobs once they age out:

Download a completed job before its TTL expires; the sweep runs on each new submission and removes the output with the job directory.

The service sends windows through the existing SeedVR2 endpoint, blends four frames at each boundary, and writes one continuous MP4 encoder. Each window is the longest 4n+1-frame clip that the sharded clip-pixel cap, the frame cap and VLLM_OMNI_SEEDVR2_LONG_MAX_WINDOW admit at the output size. Every window pays a fixed round trip through the serving stack and recomputes its overlap, and a window of four latent frames gets no temporal attention in the DiT, so raise the clip cap as far as the device allows. Windows carry source pixels, which the model resizes on the device, and cross the endpoint losslessly; only the final MP4 is compressed. The next window is restored while the previous one is blended and encoded. It repeats source frames and audio only when loop_input=true. The sharded serving profile above (VAE tiling, VAE height sharding, Ulysses degree of at least four) is required. This route keeps a window within the whole-clip pixel budget; it does not raise the /v1/videos/sync 257-frame or pixel limits. The long route uses fixed one-step, CFG=1 conditioning and requires imageio[ffmpeg] for audio muxing. The 7,200-frame 768×1344 case is still undergoing full GPU E2E.

Temporal and spatial VAE tiling¶

Add --vae-use-tiling to bound VAE intermediate activations along time. The encoder processes nine frames first, then eight per chunk; the decoder processes two latent frames per chunk. Each causal convolution carries its past inputs within that encode/decode call, with temporal stride and first-frame upsampling alignment preserved. GroupNorm and attention still see the entire spatial frame. There is no overlap blend or independent restart at chunk boundaries.

Five-frame clips use one temporal chunk. On a single GPU, convolutions with more than 256 input rows additionally process tiles of 128 output rows with exact halos. GroupNorm and bottleneck attention retain full-frame statistics. This bounds convolution workspace; full-frame activations and the DiT's whole-clip activations still consume memory.

With VAE patch parallelism, each rank owns a band of latent rows and the corresponding encoder/decoder rows. Neighbor exchanges supply convolution halos; all-reduced centered statistics preserve full-frame GroupNorm. Bottleneck attention gathers the low-resolution frame before splitting it again. Encoder parameters are gathered before latent sampling, and decoded pixels are gathered before returning the video. Unequal bands are supported; fewer latent rows than ranks fall back to replicated execution. Width sharding and batch slicing are unsupported. FP16 reduction and convolution order can change rounding, so parallel outputs are numerically close rather than bitwise identical.

Current integration scope¶

Unsupported engine modes are rejected before process hooks and worker creation. ulysses_degree selects the model-owned SP group. This branch assigns whole windows to ranks; the specialized head-sharded path is a dependent change. Ring and AllGather-KV are unsupported.

The practical P0 reference's five-frame batching, overlap, and CPU swapping are separate execution semantics. Temporal tiling and VAE patch parallelism reduce activation peaks but do not bound the whole-clip DiT memory. Large output frames may still exceed device capacity.

Invalid inputs return 400. A single-GPU model OOM fails that request; a subsequent short request succeeds. With VAE patch parallelism, restart the service after an OOM: another rank may still be waiting in a collective. Cancelling an in-progress video job removes it through the existing DELETE endpoint; in-flight GPU work may drain before memory is reusable. A lost SP worker fails the request and makes health return 503. Restart the service to restore availability; automatic rank recovery is not provided.

Reproducible deployment¶

See the RTX 5090 recipe for the input/output contract, complete serving command, and media checks. The local tests in tests/diffusion/models/seedvr2/test_seedvr2_e2e.py exercise the full 3B pipeline, all colour modes, and HTTP restoration with USP 1 and 8.