C2R# · Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision

Abstract

Coarse-to-real (C2R) video generation converts coarse animation, simulation, or game-rendered videos into photorealistic videos while preserving scene layout, camera motion, and dynamics. Training such models requires aligned (coarse, realistic) video pairs, which are expensive to create and confine prior systems to narrow synthetic domains.

We exploit a simple asymmetry: paired videos are hard to obtain, but paired real-to-coarse images are easy to construct with existing image editing and reconstruction tools. From such image pairs we train the DINO-R2C model, which transfers realistic images to coarse images in DINO feature space, and apply it to every frame of ordinary real videos to synthesize pseudo-coarse videos. The C2R# model then learns to generate each real video from its pseudo-coarse counterpart, without any authored (coarse, realistic) video pairs. At inference, the DINO-R2C model is discarded and actual coarse videos drive the C2R# model directly. Training thus scales with real-video corpora rather than with paired synthetic videos.

Method

C2R# is trained without paired coarse and real videos. First, the DINO-R2C model learns from paired realistic and coarse images and is then applied to each frame of real videos, turning them into pseudo-coarse videos. The C2R# model is trained on these pseudo-paired videos: trainable control and fusion adapters feed the pseudo-coarse video into a frozen Wan 2.2 backbone, which learns to produce the original real video. At inference, an actual coarse video conditions C2R# directly and DINO-R2C is no longer needed. Click the figure to enlarge it.

Top: the DINO-R2C bridge is trained on paired realistic and coarse images with feature and pixel reconstruction losses, then applied to each frame of a source video to synthesize pseudo-coarse videos in four styles (Coarser Geom., Coarse Geom., Coarser Anime, Coarse Anime). Bottom left: C2R# training, where the pseudo-coarse video is encoded by the frozen DINOv3 encoder and passed through trainable control and fusion adapters into a frozen Wan 2.2 model, with the source real video as the target. Bottom right: at inference, a coarse video and a text prompt condition the same model in a denoising loop. — **Top:** The DINO-R2C model is trained from paired realistic and coarse images using feature reconstruction and pixel-space reconstruction losses. Once trained, it is applied to each frame of real videos to synthesize pseudo-coarse videos. **Bottom left:** The C2R# model is trained on these pseudo-paired videos. The pseudo-coarse video is encoded by the frozen DINO encoder E and injected into the frozen Wan2.2 backbone through trainable control and fusion adapters, while the original real video provides the training target. **Bottom right:** at inference, an actual coarse video is encoded directly by E and used to condition the same C2R# model; the DINO-R2C model is no longer required. Orange modules are trainable, blue modules are frozen, gray boxes are tensors, and red boxes denote training losses.

DINO-R2C training data

Ten training examples. Each shows a photorealistic image (left) and its four coarse targets. Click an image to enlarge it.

DINO-R2C video results

DINO-R2C applied frame by frame to twenty photorealistic videos. Each video shows the photorealistic input video followed by the four coarse styles.

C2R# results

Twenty examples for each input type. In every video the input is on the left and the C2R# result on the right.

C2R#-I2V results

C2R#-I2V is a separate version of C2R# that also takes a reference image to guide appearance, in addition to the input video and prompt. A frozen Wan VAE encodes the reference image into extra tokens, and rank-16 attention LoRA adds the guidance while keeping the control and fusion adapters. Each example shows the reference image, the input and the result.

Decoding and re-encoding ablation

The features predicted by DINO-R2C already live in DINO space and could condition C2R# directly. However, they keep hidden cues of the real input video that are absent when an actual coarse video is encoded at inference, a mismatch between training and inference. C2R# therefore decodes each prediction to coarse pixels with the frozen RAE decoder and re-encodes those pixels with the same frozen encoder used at inference. Below, for the same input and prompt: C2R#, and a variant trained without re-encoding (C2R# w/o Re-encode), which conditions directly on the predicted features and shows visible artifacts.

Comparisons

Five examples for each input type, generated from the same input and prompt. Top row: input, C2R# (ours), C2R Original and Editto Global. Bottom row: Editto Sim2Real, Editto Sim2Real (rep.), Wan 2.2 VACE and Wan 2.2 VACE (rep.). "(rep.)" marks our reproductions: the same baselines fine-tuned on our pseudo-paired videos.

In these examples, C2R# (ours) produces more photorealistic results than the baselines while better preserving the structure and motion of the input. C2R Original often changes the scene, Editto Global restyles colors and lighting, Editto Sim2Real often produces misaligned, overly bright outputs that are close to white, and Wan 2.2 VACE tends to keep the input's style, which makes larger domain jumps hard, so it struggles more when the input is very coarse. Training Editto Sim2Real and Wan 2.2 VACE on our pseudo-paired videos (rep.) improves their results in several cases.

Comparison with Seedance 2.5

Two examples for each input type, generated from the same input and prompt. Left: input. Middle: C2R# (ours). Right: Seedance 2.5.

Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision