Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes

Zurich (top right)

A modern video of Zurich in the early morning, with light snowfall under a pale winter sky. Streets are partially covered with fresh snow, footsteps visible on sidewalks. Few pedestrians walk calmly, wearing thick winter coats, knit hats, scarves, gloves, boots, mostly in neutral colors. Clean, well-maintained buildings with stone façades, sloped roofs, and large windows line the street. Trams move slowly along snow-dusted tracks. The soundscape is quiet and muffled, with soft ambient city noise. The lighting is cool and diffused, emphasizing a calm, serene Winter modo.

Paris (bottom left)

A cinematic video of Paris at dawn, filmed at street level near a Haussmann boulevard. The sky is pale blue and grey, with light rain falling and wet pavement reflecting warm streetlights. Few pedestrians walk calmly, some holding umbrellas, wearing long wool coats, scarves, trench coats, leather shoes, and muted neutral colors. A cyclist passes by wearing a rain jacket. Haussmann-style buildings with cream stone façades, wrought-iron balconies, tall windows, and mansard roofs line the street. Café terraces are closed, chairs stacked neatly. Subtle traffic in the background, soft ambient city sounds, realistic motion blur and natural lighting.

Nairobi (bottom right)

A modern video of Nairobi during the dry season in the early morning. The sky is pale blue with a light haze, and the air appears dry. Streets are dusty in some areas, with patches of worn asphalt and concrete sidewalks. Pedestrians, mostly Black people, walk at a normal pace, wearing practical everyday clothing such as T-shirts, light shirts, jeans, skirts, and simple jackets, suited to mild morning temperatures. Some carry backpacks or bags. Low-rise commercial buildings, apartment blocks, and small street shops line the road, showing signs of daily use rather than perfection. Matatus, motorcycles, and cars move steadily through traffic, stopping frequently. Trees and sparse greenery appear along the streets, slightly dry. The soundscape includes engine noise, footsteps, quiet conversations, and distant city activity. The lighting is natural and flat, with soft shadows, emphasizing an ordinary, lived-in morning atmosphere.

Abstract

Traditional rendering pipelines rely on complex assets, accurate materials and lighting, and substantial computational resources to produce realistic imagery, yet they still face challenges in scalability and realism for populated dynamic scenes. We present C2R (Coarse-to-Real), a generative rendering framework that synthesizes real-style urban crowd videos from coarse 3D simulations. Our approach uses coarse 3D renderings to explicitly control scene layout, camera motion, and human trajectories, while a learned neural renderer generates realistic appearance, lighting, and fine-scale dynamics guided by text prompts.

To overcome the lack of paired training data between coarse simulations and real videos, we adopt a two-phase mixed CG–real training strategy that learns a strong generative prior from large-scale real footage and introduces controllability through shared implicit spatio-temporal features across domains. The resulting system supports coarse-to-fine control, generalizes across diverse CG and game inputs, and produces temporally consistent, controllable, and realistic urban scene videos from minimal 3D input.

Results

Our method generates photorealistic urban crowd videos from coarse 3D simulations across different scenes. For each of the scenes below, the top left video is the coarse driving input signal.

Scene: Change Cities and Weather

London (top right)

A cinematic video of London in the evening, filmed near a modern financial district street. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry. Heavy rain falls steadily, creating reflections on dark asphalt and sidewalks. Moderate crowds move briskly, people holding umbrellas, some hurrying. Pedestrians wear dark trench coats, wool coats, suits, scarves, waterproof jackets, and leather shoes. Red double-decker buses and black taxis pass by. Modern and historic buildings mix together: glass office towers alongside stone façades with large windows. Streetlights and traffic signals glow warmly through the rain. The camera slowly tracks forward, capturing water splashes, reflections, and realistic motion blur.

Shanghai (bottom left)

A modern video of Shanghai in the early evening, under overcast skies with soft diffused light. High pedestrian density on a busy commercial street. People wear modern city attire: long coats, office wear, smart casual outfits, sneakers and dress shoes, many carrying bags or using smartphones. Tall glass skyscrapers and modern commercial buildings dominate the skyline, with LED signage beginning to glow. Reflections appear on glass surfaces. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry. The camera moves forward through the crowd, capturing urban energy, realistic crowd motion, and subtle ambient city noise.

Tokyo (bottom right)

A nighttime video of Tokyo in Shibuya, under clear skies with bright neon signage illuminating the streets. Dense crowds cross a large intersection, people moving in all directions. Pedestrians wear modern urban fashion: oversized jackets, hoodies, techwear, sneakers, backpacks, business suits, and long coats. Bright LED screens reflect on glass and asphalt. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry. The camera slowly pans, capturing the constant flow of people, taxis, and buses, with vivid colors, sharp reflections, and smooth cinematic motion.

Paris (top right)

A cinematic video of Paris at dawn, filmed at street level near a Haussmann boulevard. The sky is pale blue and grey, with light rain falling and wet pavement reflecting warm streetlights. Few pedestrians walk calmly, some holding umbrellas, wearing long wool coats, scarves, trench coats, leather shoes, and muted neutral colors. A cyclist passes by wearing a rain jacket. Haussmann-style buildings with cream stone façades, wrought-iron balconies, tall windows, and mansard roofs line the street. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry. Café terraces are closed, chairs stacked neatly. Subtle traffic in the background, soft ambient city sounds, realistic motion blur and natural lighting.

Zurich (bottom left)

A modern video of Zurich in the early morning, with light snowfall under a pale winter sky. Streets are partially covered with fresh snow, footsteps visible on sidewalks. Few pedestrians walk calmly, wearing thick winter coats, knit hats, scarves, gloves, boots, mostly in neutral colors. Clean, well-maintained buildings with stone façades, sloped roofs, and large windows line the street. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry. Trams move slowly along snow-dusted tracks. The soundscape is quiet and muffled, with soft ambient city noise. The lighting is cool and diffused, emphasizing a calm, serene winter mood.

Sunny (bottom right)

People crossing a street, in a city, the day is clear and sunny, the streets are busy and there are different vehicles in the streets. Foreground building is a long horizontal block with deep recessed horizontal bands. Background features vertical skyscrapers with simple grid geometry, the streets are clean and people are wearing different types of clothes and styles suitable for warm summer days.

Scene: Add Landmarks

Use text prompts to add landmarks like Eiffel Tower or Oriental Pearl TV Tower to the scene in the background.

Paris, Eiffel Tower

A realistic, high-quality video set on a wide Parisian city street, surrounded by elegant mid-rise buildings with classic European façades forming an urban corridor. Two people enter from the lower-left foreground and walk forward side by side at a steady pace, while the camera smoothly follows behind them at pedestrian height, maintaining a stable forward-tracking motion. On the right side of the street, close to a building wall, two other people stand facing each other, actively talking and arguing with visible hand gestures and expressive body language. Pedestrians wear autumn clothing such as coats, jackets, dresses, scarves, and boots, suited to mild fall weather. Far ahead in the distance, another solitary person stands or walks slowly, appearing small due to depth and perspective. In the far background, framed between the buildings, the Eiffel Tower rises prominently on the skyline as a recognizable landmark. Human motion is natural and continuous, with realistic walking cycles and subtle gestures, while the camera movement remains smooth and stable. The scene is lit with soft natural daylight, clear depth, and balanced exposure, presented in a cinematic, photorealistic style with sharp details, clean textures, and no blur or visual artifacts.

Shanghai, Oriental Pearl TV Tower

A realistic, high-quality video set on a wide city street in Shanghai, surrounded by tall modern buildings forming a dense urban corridor. Two people enter from the lower-left foreground and walk forward side by side at a steady pace, while the camera smoothly follows behind them at pedestrian height, maintaining a stable forward-tracking motion. On the right side of the street, close to a building wall, two other people stand facing each other, actively talking and arguing with expressive hand gestures and animated body language. Far ahead in the distance, another solitary person appears small in the frame, standing or walking slowly. In the far background, visible between the buildings, the Oriental Pearl TV Tower rises prominently on the skyline as a recognizable Shanghai landmark. Human motion is natural and continuous, with realistic walking cycles and subtle gestures, and the camera movement remains smooth and consistent. The scene is lit with soft natural daylight and light urban haze, featuring clear depth, balanced exposure, and a cinematic, photorealistic style with sharp details, clean textures, and no blur or visual artifacts.

Scene: New York on A Different Day and Time

Use text prompts to change the time and weather in New York.

Coarsening Level Adaptation

Our method is compatible with different coarsening levels of input signal. On the left column we show results for very coarse signal, where our model inpaints many details. On the right we show a mid-coarse geometry as input that already includes some fine details. Our model successfully follows the richer input signal, generating a video with the corresponding geometry.

Applications

Our model can turn a low-poly game video into real-style even though it has never been trained on any game videoss. It can add realism to the existing dynamics and appearance, add extra contents and solve collision issues in the original video.

Comparison

Comparison with WAN2.1 + ControlNet

We compare our results with a baseline trained with off-the-shelf ControlNet adaptor for Wan2.1. Our model yields more expressive videos, where both humans and scene exhibit more realistic appearance.

We also compare to off-the-shelf Wan2.1 using text-to-video directly.

Off-the-shelf and Commercial Models

WAN2.1 (left column) and SORA (right column) show limited controllability over human motion and camera trajectories, and tend to generate similar viewing angles across populated urban scenes.