Modern generative AI is hitting a familiar wall: every time we try to make models “smarter” by stuffing in more knowledge, we also make them more expensive to run. In classic dense Transformers, memory and compute are tightly coupled, more parameters mean more FLOPs and memory, both during training and inference. To break this coupling, researchers have turned to conditional and augmented computation. Two of the most important approaches are Mixture of Experts (MoE) and the newer Mixture of Value Embeddings (MoVE). They attack the same problem from very different angles.
MoE scales models by adding many specialized subnetworks, called experts, while only activating a small subset of them for each token. A routing network decides which experts are used, typically selecting just one or two. This allows models to grow to hundreds of billions of parameters without paying the full compute cost every time. In practice, MoE replaces the feed-forward layers in Transformers and relies on sparse activation to keep inference affordable. The upside is massive total capacity; the downside is complexity. Routing introduces engineering challenges like expert capacity limits, dropped tokens, and heavy all-to-all communication across GPUs. Training is also fragile: if routing becomes imbalanced, a few experts get all the traffic while others “die,” forcing the use of auxiliary load-balancing tricks that can hurt final model quality.
MoVE takes a more surgical approach. Instead of adding subnetworks, it augments the attention mechanism itself. Research into interpretability has shown that the Value stream of attention carries much of a Transformer’s semantic content. MoVE exploits this by introducing a global bank of learnable value embeddings, concept vectors shared across all layers. At each token step, the model softly mixes a subset of these vectors and adds them to the standard value projection. The key idea is that you can scale memory simply by increasing the size of this embedding bank, without deepening the network or increasing active FLOPs. Memory and compute become genuinely decoupled.
These architectural differences lead to very different scaling behaviors. MoE sparsely couples capacity to compute: total parameters grow, but only a fraction are active per token. MoVE fully decouples them: compute stays constant while memory grows along an independent axis. As a result, MoVE can enter “memory-dense” regimes where the model stores far more static knowledge without becoming slower at reasoning.
Optimization dynamics differ just as sharply. MoE’s hard routing makes it vulnerable to expert collapse and requires careful balancing. MoVE uses soft, differentiable mixing, so every memory slot receives gradients and none are starved. The challenge there is not collapse but selective, ensuring the memory bank doesn’t blur into a generic average.
Finally, both approaches pay off. In text generation, MoVE consistently improves perplexity as memory slots scale, suggesting that factual knowledge can be offloaded into the embedding bank while attention layers focus on structure and reasoning. In autoregressive image generation, MoVE boosts visual fidelity by acting as a shared visual memory, and it integrates cleanly with low-rank attention schemes that are already optimized for efficiency. MoE, meanwhile, continues to shine when domain specialization matters, with different experts naturally capturing syntax, programming patterns, or language-specific features.
From a systems perspective, the trade-offs are stark. MoE places heavy demands on networking and memory bandwidth because of token routing across devices, making deployment challenging without high-end interconnects. MoVE behaves much more like a standard Transformer, with lower communication overhead and simpler scaling. Both can benefit from offloading or compression, but MoVE’s shared memory has better locality than sparsely accessed experts.
Zooming out, MoE and MoVE sit within a broader movement toward memory-augmented and modular neural architectures. Other ideas, like product-key memories, parameter-efficient expert retrieval, or even more radical geometric memory systems, share the same goal: separating “where knowledge lives” from “how reasoning happens.” In multimodal systems, similar mixture principles already show clear gains by routing inputs to specialized encoders.
The most likely future is not a winner-takes-all scenario but hybrids. MoE layers can handle specialized reasoning and task-specific logic, while MoVE augments attention with a global, editable knowledge store – like a book for lookups. Combined with efficient attention compression, such systems promise models that are not just larger, but smarter per FLOP.