LTX Trainer Is Here: Fine-Tune an Open Audio-Video Model on Your Own Data

TL;DR

Lightricks open-sourced LTX Trainer, the official fine-tuning framework for LTX-2, their DiT-based audio-video foundation model. It lets you teach a 22B-parameter video model your own characters, style, motion, and IP by training LoRAs and IC-LoRAs on your data, on your hardware. It ships with 13+ conditioning modes out of the box (text/image-to-video, audio-to-video, video-to-audio Foley, inpainting, outpainting, extension, and reference-based IC-LoRAs), an agentic setup you can drive from Claude Code, and a low-VRAM config that trains on a single 32GB consumer GPU like an RTX 5090. The whole thing is open source and maintained by the core LTX team. Your data and weights never leave your infrastructure.

It Was Never About Prompting. It Was About Control.

The launch film makes the pitch better than a spec sheet could: you prompt, sure, but prompting was always a way of asking a closed model nicely. You hand it the wheel and call that creating. The closed labs built one model for everyone, so it understood no one in particular. Not your characters. Not your style. Not the way you work. At some point a lot of people just stopped fighting it.

LTX Trainer flips the relationship. The base model is a starting point, not the product. What you build on top of it is the product, and it is yours: you own the weights, you keep the data, and the adapter you train reproduces your world instead of a generic borrowed one. That is the entire reason an open audio-video model with a real trainer matters more than another point bump on a closed leaderboard.

"The model is the starting point. What you build on top of it is what makes it yours."

What LTX Trainer Actually Does

LTX-2 is, in Lightricks' framing, the first DiT-based audio-video foundation model: it generates synchronized sound and picture together, not video with a soundtrack bolted on later. The trainer exposes that full surface for fine-tuning. The headline capabilities:

• One framework, 13+ conditioning modes. Text-to-video, image-to-video, audio-to-video, video-to-audio (Foley), text-to-audio, video and audio extension, inpainting and outpainting, keyframe interpolation, and region regeneration. They all run through a single unified conditioning architecture, so you are not learning a new tool per task.
• Combine conditioning methods. Because it is one architecture, you can mix modes: train a joint audio plus video adapter with cross-modal attention, or freeze one modality and condition on it.
• Three training approaches. Parameter-efficient LoRA (small portable adapters), full fine-tuning (distributed with FSDP for big updates), and IC-LoRA (in-context / reference LoRAs) for video-to-video, depth, pose, and structural control.
• Agentic setup. You can configure and launch runs with Claude or any LLM instead of hand-editing YAML (more on that below).
• Fully open source, maintained by the core LTX team, with a Discord for help and result sharing.

LoRA vs IC-LoRA vs Full Fine-Tune (Pick the Right One)

Most people reach for the wrong knob. Here is the quick decision guide:

• Style LoRA: teaches an aesthetic, color, texture, lighting, a look. Best for "everything I generate should look like my brand."
• Motion LoRA: teaches a movement pattern, a dolly-in, a specific camera push, an object rotation. Trained on short clips, not stills.
• IC-LoRA (reference / in-context): teaches structural control. Feed it a depth map, pose, or edge guide and it locks composition or camera path while you change the visual content on top. This is the one for "keep the layout, swap the world."
• Full fine-tune: only when a LoRA genuinely cannot capture what you need. It is heavier, needs FSDP across GPUs, and is overkill for most style or character work.

Trained LoRA adapters are small, portable .safetensors files that load alongside the base model at inference. You can stack up to three at once, so a style adapter plus a motion adapter plus an IC-LoRA can run together for a combined effect.

System Requirements (Read This Before You Get Excited)

This is a 22B model. It is not gentle on hardware, but the low-VRAM path is genuinely usable on a high-end consumer card.

• OS: Linux. The trainer depends on Triton, which is Linux-only. There is no clean native Windows path; use WSL2 or a Linux box.
• CUDA: version 13+ recommended.
• GPU (standard config): an Nvidia card with 80GB+ VRAM (H100-class) for the full-fat configuration.
• GPU (low-VRAM config): a single 32GB card such as an RTX 5090, using the provided low-VRAM YAML that turns on INT8 quantization, an 8-bit optimizer, gradient checkpointing, and reduced rank.
• Model files: a local LTX-2 checkpoint (for example ltx-2.3-22b-dev.safetensors or the distilled variant) plus the Gemma 3 text encoder directory.

Two rules the model bakes in that will save you a failed run: frame counts follow the 8n+1 rule (1, 9, 17, 25, 49 frames), and width and height must each be divisible by 32.

How to Train Your Own: The Manual Path

The flow is install, prepare data, preprocess, train, then load the adapter at inference. Concretely:

1. Install.

git clone https://github.com/Lightricks/LTX-2
cd LTX-2
uv sync
cd packages/ltx-trainer

2. Prepare your dataset. Collect your clips or images and write captions for each, then point a dataset.json at them (the repo's Dataset Preparation guide documents the exact JSON schema). For a style LoRA, 20 to 50 images is usually enough, push to 80 to 120 for a very specific subject. For a motion LoRA, use short, consistently framed video clips instead of stills.

3. Preprocess into latents. This bakes your media into the buckets the model trains on. Audio latents are pulled from your videos automatically.

uv run python scripts/process_dataset.py dataset.json     --resolution-buckets "960x544x49"     --model-path /path/to/ltx-2-model.safetensors     --text-encoder-path /path/to/gemma-model

4. Launch training. Single GPU:

uv run python scripts/train.py configs/t2v_lora.yaml

On a 32GB card, swap in the low-VRAM config instead (for example t2v_lora_low_vram.yaml). For multiple GPUs:

uv run accelerate launch scripts/train.py configs/t2v_lora.yaml

5. Sample early, stop early. Generate a test clip at checkpoint 500. If it already looks right, stop; do not burn hours chasing diminishing returns. Field reports put a mid-sized LoRA at roughly 3 to 5 hours on a single high-end card.

Settings that actually matter, per the practitioners: rank 32 is the right default (rank 64 barely moved the needle on complex sets), and learning rate 1e-4 is the reliable starting point.

How to Train Your Own: The Agentic Path

The part that fits this site's whole thesis: you do not have to hand-run any of that. The repo ships a Claude Code skill at .claude/skills/train-model. Open the repo in Claude Code and run:

/train-model

The skill does an end-to-end guided run: it probes your data and your hardware, picks the matching training mode, prepares the dataset, launches the job, and monitors it, pausing for your approval before any heavy work. It is the difference between reading three docs to figure out which config your 5090 can survive and just letting an agent inspect the box and decide. This works with any capable LLM, but it is built to be driven from Claude Code.

Real-World Examples Worth Stealing

• A consistent brand character. Train a LoRA on 40 to 80 frames of your mascot or spokesperson so it shows up on-model in every ad, instead of re-rolling a closed model and praying the face matches.
• Product photography in motion. A style LoRA on your studio look (white seamless, your lighting, your color grade) so every product video matches your catalog without a shoot.
• A signature camera move. A motion LoRA on a 15-second dolly-in at consistent framing, then reuse that exact push across an entire campaign.
• Layout-locked variants with IC-LoRA. Feed a depth or pose guide to keep the composition and camera path fixed while you swap the scene's content, perfect for A/B creative or localized versions.
• Synced sound, not afterthought audio. Because LTX-2 is audio-video native, you can train video-to-audio (Foley) or joint adapters so the generated sound actually tracks the picture, including lip dubbing and speaker matching.

Why This One Matters

Open weights without a real trainer are a museum piece. A trainer without open weights is a rental. LTX Trainer is both halves: a capable open audio-video model and the official, maintained machinery to bend it to your own data, on hardware a serious hobbyist can actually own. The closed labs optimize for the average of everyone. This optimizes for you. That is the whole game.

Key Takeaways

• LTX Trainer is the open-source fine-tuning framework for Lightricks' 22B LTX-2 audio-video model, maintained by the core team.
• It supports LoRA, IC-LoRA, and full fine-tuning across 13+ conditioning modes (video, audio, cross-modal, reference) from one architecture.
• 80GB VRAM for the standard config, but a low-VRAM path trains on a single 32GB card (RTX 5090) via INT8 quantization. Linux plus CUDA 13+ required.
• Manual flow: uv sync, build a dataset.json with captions, process_dataset.py, then train.py with a config. Sample at checkpoint 500, rank 32, LR 1e-4.
• Agentic flow: run /train-model in Claude Code and let it probe your hardware, pick the mode, and run the job with approval gates.
• Your data and your weights never leave your infrastructure. You own the result.

Sources: LTX-2 on GitHub, LTX Trainer README, LTX-2 Training docs, LTX Trainer Quick Start, Lightricks on Hugging Face, WaveSpeed LTX-2.3 LoRA Training Guide.