SANA-Video × Tom & Jerry Research Log

Key Hyperparameters

Class Prompt

The first run uses a single class prompt as the main style anchor for the whole training:

A vintage slapstick 2D cartoon scene of a grey cat chasing a small brown mouse in a colorful house, Tom and Jerry style, bold outlines, limited color palette, exaggerated expressions, smooth character motion.

Checkpoint Progression (Same Seed & Prompt)

Every few steps, a clip is generated with a fixed seed and CFG. This makes it easy to see how the LoRA gradually steers SANA-Video from its generic prior into the Tom & Jerry regime.

Step 100

Still messy; style only slightly nudged away from the base.

Step 1k

Clearer outlines, more cartoony color palette starting to appear.

Step 5k

Motion and silhouettes look much closer to classic TV-era animation; Tom is a lot easier to read.

Step 10k

Style mostly locked in; further training starts to trade diversity for fidelity.

Text-Conditioned Experiments

After the class-LoRA run converges, the next question is: how much text-conditioning power is left? To test this, we reuse the same LoRA weights but vary the prompts.

All of the clips below share:

• The same LoRA checkpoint.
• The same seed (32420).
• Identical sampling settings (steps, CFG, etc.).

Class baseline

Same style anchor as the training prompt; sanity check for the LoRA behavior.

Night living room

Tests lighting changes and sneaking motion in a cozy living room.

Vertical staircase chase

Same style, different geometry and motion (up/down motion in an old mansion stairwell).

Tom eating pizza

Adds an explicit object + action (pizza) on top of the base tom and jerry prompt setup.

Backyard prank

Moves the scene outdoors to a sunny backyard with slapstick prank dynamics.

V0: CFG Sweep Evaluation

I know what you thinking... what are those flashes and blurines? until know while generating we used a set cfg = 6.0 which cuses theose weird distorations, usinge a 100 ground-truth Tom & Jerry segments we will generate 40 corresponding clips with the V1 LoRA at different CFG scales but same list of seeds, and compare each generated clip to its ground-truth counterpart.

For each CFG setting, we compute:

• SSIM ↑ – structural similarity to the ground-truth frames.
• PSNR ↑ – per-frame signal-to-noise ratio.
• LPIPS ↓ – perceptual distance (AlexNet or VGG backbone).
• FID ↓ – frame-wise FID over all frames (Inception-V3 features).
• tLPIPS ↓ – temporal LPIPS between consecutive frames (motion smoothness).

V2: Scene Dataset & Training

V1 was intentionally simple: one global class prompt and a LoRA that “pulls” SANA-Video into the Tom & Jerry style at 224×224. In V2, the goal is to move beyond a single prompt while preserving as much control from the foundation model as possible, and to build a richer, text-aware scene dataset that the model can actually react to.

Dataset creation

The idea is to extend the existing Tom & Jerry latent cache with per-clip scene descriptions. Using recent advances in multi-modal, affordable models and efficient inference pipelines (continuous batching, KV-cache, vLLM in practice for more deatils explore this blog ), labeling a large dataset with detailed scene descriptions becomes realistic with reasonable engineering effort:

• Start from the same curated clips used in V1 (already resized & cached as latents).
• Use Qwen3-VL-8B served via vLLM to auto-label each segment.
• Add a CSV/JSON file that stores one text_prompt per clip or segment.
• Encode these text prompts once and cache the embeddings alongside the video latents.

Scene description template & labeling prompt

Each segment is described with a structured scene card (environment, characters, props, action, camera). The same template is used as the base for all prompts fed into the VLM.

You are an expert animation director and prompt engineer.

You are given a short video segment (about 3–6 seconds) showing a Tom and Jerry cartoon scene.
Your job is to:
1) Understand the scene (environment, characters, actions, style, camera).
2) Return a structured description that can be used as a base prompt for a text-to-video model.

Follow this exact output format and headings, and DO NOT add any extra sections:

ENVIRONMENT:
- Describe the location and setting (e.g. "inside a colorful living room", "in a sunny backyard").
- Mention key background elements (walls, furniture, sky, ground, etc.).
- Include mood / lighting (e.g. "warm lamp light", "bright midday sun", "dark night with moonlight").

CHARACTERS:
- List each main character separately.
- For each: species/type, color, size, clothing/accessories, personality vibe.
  (e.g. "a grey cat with white paws, energetic and mischievous").
- If the character is Tom or Jerry, mention their names explicitly ("Tom", "Jerry").

PROPS:
- List important objects that are involved in the action or stand out visually.
  (e.g. "sofa, table, pizza slice, ladder, flower pot").

ACTION:
- Describe what happens in the segment in 1–3 short sentences.
- Focus on interactions between characters and props, and the main motion.
  (e.g. "the cat chases the mouse around the table and almost slips on a rug").

CAMERA & FRAMING:
- Describe the main camera view (e.g. "side view, medium-wide shot").
- Mention if the camera is static or moving (panning, zooming, following the character, etc.).

Write clearly, concisely, and keep everything in this exact structure.

V2: Dataset Labeling Failure Modes

Qwen3-VL is a very strong VLM, but when I used it to label the Tom & Jerry dataset, several systematic failure modes kept showing up(it still mostly labeled correctly the scenes but I did notice those failures casses). Below are four representative cases, each showing the generated scene description on the left and the corresponding video on the right.

Future Work

This is a living project and there are a bunch of directions I want to explore next. The bullets below are placeholders and will be updated as the project evolves.

• Resolution × CFG benchmarks: refine the evaluation protocol and run a full sweep across CFG scales, comparing V1 vs V2 side-by-side on SSIM, FID, LPIPS, and temporal metrics.
• Motion-score conditioning: experiment with adding a motion score (e.g. Unimatch + VMAF, similar to how SANA-Video scores motion) as an extra token appended to the text conditioning, and train the model to respond to “more / less motion” controls.
• RL-style reward model: take inspiration from Video Generation Models Are Good Latent Reward Models , clone a pruned copy of the video transformer as a latent reward model that scores “real vs fake” Tom & Jerry clips, and use those scores in a lightweight RL loop to sharpen dynamics and reduce artifacts.