Training Text-to-Image Models 3. 6× Faster

Training Text-to-Image Models 3. 6× Faster. IntroductionContents×IntroductionHitting the VAE compression wallUnlocking aggressive compression with a unified modelExtending JiT for text-to-image modelsRecovering finegrained details in pixel-spaceArchitecture refinementsFrom Linum v2 to JiT-DDTWhy does the JiT-DDT work?

What happened

But if you take a survey of the popular open source text-to-image models like FLUX, Ideogram, and Z-Image, you'll notice that they all cap out at 16×16 token reduction. Extending JiT for text-to-image models When we read about JiT, we were really excited to give it a go, since it was explicitly able to achieve 32×32 token reduction. Attention is quadratic in cost, so the biggest lever we have to accelerate model training is pruning the context window down.

Most generative image and video systems are Latent Diffusion Models (LDMs). Against our Linum v2 baseline, the JiT-DDT trains a text-to-image model with 3. Hitting the VAE compression wall Almost all generative image and video models are Latent Diffusion Models (LDMs). The VAE is trained once and frozen;the DiT learns to move through its latent spaceTrainingInferencereplayinput x❄encoder= E(x)z(1−t)·z+ t·εINTERPOLATEzₜDiTtrainableditv̂pred‖v̂ − v‖²v = ε − zlossε ~ N(0, I)gaussianεSAMPLE εt ~ LogitNormalsample t‹ready›VAE frozen (dashed) · DiT trainable (purple) · gradient stops at the DiT · t = 0 clean image, t = 1 pure Gaussian noise We want to eke out as much token compression as possible from the VAE, so that we can curb the cost of attention in our DiT.

The wider picture

If we believe that images and videos naturally lie on a low dimensional manifold, we should be able to have our models predict effectively even with high. And on toy benchmarks like ImageNet, LDMs still out-perform pixel space models by a smidge. Text-to-image baselines When we pretrained Linum v2, we relied on a VAE + patchification stack that afforded 16×16 token reduction. But, we also made a slew of other tweaks to the network: One single-stream DiT does the compressing and the generatinggray = frozen · dashed = loss concat · add · multiplySingle stream backboneInstead of alternating blocks of self-attention (image/video) and cross-attention (text-to-image/video), we concatenate visual tokens and text tokens into a single stream that goes through the DiT.

This increases the attention sequence in every block and increases the FLOPs per token, but should allow for significantly more expressive relationships between text and image tokens. v2 block: self-attention, then cross-attentionv3 block: one self-attention over image and textWider instead of deeperOur old model was a 40-layer transformer with 2048 hidden size. Appendix TL;DRLinum v2 was bottlenecked by the enormous size of its attention context window. To put that in perspective, LLMs see samples with fewer than 8K tokens for 97% of their pretraining.

Recently, pixel-space models like the JiT have shown to be a promising alternative.

What has been reported

It reduces two models into one and allows the diffusion model to construct a latent space specifically for generation, rather than rely on one built for reconstruction. When trained on our (image, caption) dataset, the JiT seems to struggle to produce finegrained details. 6× fewer GPU-hours, even though it generates images with 4× the pixels. 6× faster to train, at 4× the pixelsLinum v2 (ours, previous)* · 256×2562. 0B latent-space DiT + VAE256 latent tokens* image-only checkpointJiT-DDT (ours, new) · 512×5122. We hope that by sharing our findings with the broader community, we can encourage others to also explore more efficient training methods.

This way the latent space stays static throughout the course of DiT training. This aligns with our experiments on Image-Video VAEs from a few years ago.

What happens next

In JiT at 512px we use 32×32 patches, so a 512×512 image becomes 256 tokens, each starting at 32·32·3 = 3,072 dims; the bottleneck maps that to 256. We used it in Linum v2 and so do models like FLUX. There are small hacks like using an external model as a regularizer during VAE training (e. g. DINOv3) that (likely) enabled models like FLUX-2 to make the leap from 64 latent dimensions to 128 latent dimensions for their DiT. Rather, it's downstream of the v-prediction, v-loss flow matching objective that everyone's been using to train diffusion models these past few years.

V-prediction and the curse of dimensionality If you're training a flow matching model you don't necessarily need to train your neural network to predict and regress velocity. For each of these spaces, they trained flow matching models with x-prediction, -prediction, and velocity-prediction; and found that the x-prediction was the only model type to accurately generate samples from the spiral distribution at large dimensions.

The report has been compiled by The Daily Waves using information reported across linum.ai, rohanbansal.com. Details are presented according to the information available at the time of publication and may change as authorities, organisers or other relevant parties provide updates.