AURA
Video and audio from one forward pass
A ~30B-parameter foundation model that natively generates 2K video and synchronised stereo audio together, in a single forward pass, instead of stitching two models in post.
- Where
- Stealth AI startup
- When
- 2026 — present
- Role
- AI Research Engineer — architecture, AV sync, inference cost
- Stack
- PyTorch · Diffusion Transformers · Rectified Flow · Latent Consistency Models · Distributed training · CUDA
Problem
Generated video and generated audio almost always come from two separate models, aligned afterwards in post. That is where lip-sync drifts, foley lands on the wrong frame, and rhythm stops matching motion — and no amount of post-production stitching fully hides it. AURA generates both modalities natively in one pass so alignment is a property of the model rather than a repair step.
Constraints
- 01Audio and video must be produced together, not stitched after the fact
- 02Millisecond-level AV alignment — lip-sync, foley and rhythm
- 03Sampling cost low enough that cinematic-length content is affordable to produce
- 04Mixed conditioning — text, reference images, motion and audio — without losing identity
Architecture
Key decisions & trade-offs
Rectified flow matching instead of standard diffusion
Flow matching with a rectified objective straightens the ODE trajectories the sampler has to follow. Straighter paths are both more stable to train and far cheaper to integrate, which is what makes an aggressive step reduction possible later rather than a wish.
Trade-offIt commits the architecture to a flow formulation, so the large body of diffusion tooling and scheduler tricks does not transfer directly. The training stability and the step budget were worth that.
A bidirectional attention bridge, not post-hoc alignment
Rather than generating each modality independently and aligning afterwards, the two branches attend to each other during generation. Lip-sync, foley and rhythm fall out of the forward pass because each branch can see what the other is doing while it is doing it.
Trade-offCross-modal attention couples the branches, so they cannot be scaled, cached or served independently, and a change to one affects the other. That coupling is the feature — it is also the cost.
Step distillation via Reflow and Latent Consistency Models
Fifty sampling steps is a research result; eight is a product. Distilling the sampler down to ≤8 steps is what moves the cost per second of footage into a range where you can actually generate cinematic content.
Trade-offFew-step samplers characteristically trade some output diversity and fine high-frequency detail for speed. For production footage under directorial control, determinism is closer to a feature than a loss.
Decoupled cross-attention injection for mixed conditioning
Text, reference images, motion and audio are injected through separate cross-attention paths rather than a single concatenated conditioning vector. That keeps identity from a reference image from being washed out by a long text prompt.
Trade-offMore attention paths mean more parameters and more places for conditioning signals to conflict. It buys the thing that matters for real use — you can change the direction without losing the character.