Four startup bets on what comes after transformer LLMs

The next generation of LLMs may be defined by architecture, not just scale. A new MIT Technology Review report describes four startup approaches to the weaknesses of transformer-based models: reducing attention cost, replacing attention, shrinking the model, generating text in parallel and reasoning beyond language. The article is a useful map of experiments—not proof that any one startup has won.

Definition: Post-transformer LLM research changes how a model stores context, computes attention, generates tokens or represents reasoning.

Example: A long-context model might retain a compact summary instead of tracking every token, while a diffusion model might generate a block of text in parallel.

Key takeaway: The important question is not which architecture sounds newest; it is which one completes a real workload with better quality, cost or latency.

Business impact: More efficient architectures could make long-context agents, local models and low-latency automation easier to operate, but each claim still needs workflow-level evaluation.

Why are transformers becoming a bottleneck?

Transformers remain the dominant architecture for large language models, but dense attention compares every token with every other token, so the computational burden rises rapidly as context grows. The MIT Technology Review report illustrates the problem with a 10,000-word document that could require roughly 50 million multiplications; for a business, the takeaway is to measure long-context cost and reliability instead of assuming a larger context window is free.

Reasoning models add another pressure because their scratchpad creates more text for the model to track, while agents may need to process repositories, document collections or outputs from other models. The practical implication is that a model can appear capable in a short prompt yet become expensive or unreliable when the workflow needs sustained context; teams should test the full context they expect to use.

Can sparse attention make long context cheaper?

Subquadratic is testing whether an LLM can keep the useful relationships in a long input without calculating every possible token pairing. The company says its SubQ model identifies which words matter for each piece of text and uses sparse attention; the business takeaway is to treat this as a long-context efficiency claim that must be checked on retrieval, coding and tool-use workloads.

Manifest AI takes a different route to the same bottleneck with power retention, which stores a rolling summary of context and drops less relevant information as new material arrives. Manifest AI says its retention layer can replace a transformer's attention layer and that a small amount of retraining can convert existing models, including a StarCoder2-3B demonstration; operators should compare summary-based memory with full-context retrieval for information loss, not only compare headline speed claims.

Can smaller hybrid models run on devices?

Liquid AI combines transformer layers with liquid neural networks to build liquid foundation models that adapt their behavior to new information and use less hardware than many conventional LLMs. The source article reports that recent LFMs use a 20% transformer and 80% liquid-neural-network mix; teams considering edge or private deployment should measure quality, memory, latency and update behavior on the exact device.

The business significance of Liquid AI's approach is deployment flexibility rather than a universal replacement for cloud models. The source says Liquid AI models are used for vehicle chips, can run on a Raspberry Pi and are available at no charge to organizations below $10 million in annual revenue; the relevant decision is whether a smaller model clears the workflow's quality bar while reducing connectivity, latency or infrastructure requirements.

Can diffusion generate text in parallel?

Inception is applying diffusion—the method widely associated with image generation—to language so that an LLM can produce larger blocks of text in parallel instead of emitting one token at a time. The source reports claims that Mercury 2 can approach some GPT-4-era performance at roughly ten times the speed; teams should validate time to useful output, not only tokens per second.

The trade-off is that text does not have the smooth intermediate states that pixels do, so diffusion language models need different training and decoding methods. Inception's founders say they developed the mathematical approach for applying diffusion to text, but the article's evidence remains company-reported; a production test should include structured output, code correctness, retries and cost per completed task.

Can LLMs reason beyond language?

Pathway argues that some problems are awkward for a model forced to represent every step as text, and it uses a state-space structure in Dragon Hatchling to support more abstract reasoning. The source reports that Dragon Hatchling beat more than 97% of a benchmark containing over 250,000 difficult Sudoku puzzles and presents the result as evidence for reasoning that is not limited to word sequences; the takeaway is to test non-linguistic tasks directly instead of inferring reasoning ability from fluent prose.

Pathway's Sudoku result is a sharp demonstration, not a general business benchmark. A model can solve a constrained puzzle and still be unsuitable for document extraction, customer support or code changes, so operators should ask whether the architecture improves the specific reasoning substrate their workflow needs and whether its outputs remain verifiable.

What should operators watch next?

These startups are pursuing different bottlenecks rather than one shared replacement for transformers: Subquadratic and Manifest AI focus on context efficiency, Liquid AI focuses on compact adaptive models, Inception focuses on parallel generation and Pathway focuses on latent reasoning. The AI models market overview already points to workflow-level testing as the practical filter; the next step is to compare these architectures on the same task, quality threshold, latency target and cost accounting.

The first useful tests should be narrow and reproducible. For a long-context agent, measure retrieval accuracy, context loss and cost; for a local model, measure memory, latency and privacy boundaries; for a diffusion model, measure time to usable output and schema compliance; for a reasoning model, measure correctness on tasks whose constraints are explicit. This is also why AI infrastructure is becoming the layer under everything rather than a single model choice.

A business should not replace a working model because a startup has a compelling architecture. It should keep a baseline, run representative cases, record failures and compare cost per successful task; running an LLM locally is one relevant option when a smaller model can meet the requirement without sending every request to the cloud.

The bottom line

The transformer era is not over, but its bottlenecks have become the startup opportunity. Sparse attention, retention, liquid networks, diffusion and latent reasoning each offer a different answer to the same question: how can LLMs become more efficient, more persistent or better suited to tasks that language alone does not represent well?

The near-term winners will be decided by evidence outside the demo. For operators, the durable advantage is a measurement loop that compares quality, cost, latency, context handling and safety on real work; for researchers, these startups are evidence that the next important LLM improvement may come from changing the architecture underneath the interface.