This startup’s new mechanistic interpretability tool lets you debug LLMs

The Black Box No More: How Goodfire’s Silico Is Turning AI Debugging Into a Science

For years, large language models (LLMs) like ChatGPT, Gemini, and Llama have dazzled the world with their ability to write poetry, code software, and even pass medical exams. Yet beneath their fluent prose and seemingly intelligent responses lies a troubling truth: no one truly understands how they work. These models are digital black boxes—massive neural networks trained on trillions of words, capable of astonishing feats but prone to hallucinations, biases, and unpredictable behavior. Fixing these flaws has often felt less like engineering and more like alchemy: add more data, tweak a few parameters, and hope for the best.

Now, a San Francisco-based startup called Goodfire is aiming to change that. With the launch of its new tool, Silico, the company claims it has built the first off-the-shelf mechanistic interpretability platform that lets developers peer inside LLMs at every stage of development—from data curation to training and deployment. The goal? To turn AI development from a guessing game into a precise, repeatable science.

The Alchemy of AI: Why LLMs Remain Mysterious

Despite their impressive capabilities, modern LLMs are fundamentally opaque. Trained on vast datasets scraped from the internet, they learn patterns through billions of interconnected artificial neurons. But unlike traditional software, where every line of code has a known function, LLMs emerge as complex systems where cause and effect are deeply entangled. When a model generates a false fact or exhibits biased reasoning, developers often can’t trace the error back to a specific neuron or data point.

This opacity has led to what many in the field call the “alchemy problem.” Just as medieval alchemists experimented with ingredients without understanding the underlying chemistry, today’s AI researchers often rely on trial and error, scaling up models in hopes that performance improves. The dominant belief in many leading AI labs is that more data, more compute, and bigger models will inevitably lead to artificial general intelligence (AGI)—a kind of technological singularity where understanding becomes unnecessary.

But Goodfire’s CEO, Eric Ho, challenges this assumption. “We saw this widening gap between how well models were understood and just how widely they were being deployed,” he told MIT Technology Review. “The dominant feeling is that if you just scale up, you’ll get AGI and nothing else matters. We’re saying no—there’s a better way.”

What Is Mechanistic Interpretability—And Why Does It Matter?

Mechanistic interpretability is a cutting-edge approach to AI that aims to reverse-engineer neural networks by mapping the roles of individual neurons and the pathways between them. Think of it as neuroscience for artificial brains: instead of treating the model as a monolithic black box, researchers dissect it to understand how specific circuits contribute to specific behaviors.

This technique has gained traction in recent years, with companies like Anthropic, OpenAI, and Google DeepMind investing heavily in interpretability research. MIT Technology Review even named mechanistic interpretability one of its 10 Breakthrough Technologies of 2026, citing its potential to make AI safer and more controllable.

Goodfire is taking this research a step further. While most interpretability work focuses on auditing already-trained models, Goodfire wants to embed understanding into the development process itself. “We want to remove the trial and error and turn training models into precision engineering,” says Ho. “That means exposing the knobs and dials so you can actually use them during training.”

Silico, the company’s new platform, automates much of this interpretability work using AI agents—software programs that can analyze model internals, identify problematic circuits, and suggest fixes. This represents a major leap: previously, mechanistic interpretability required teams of PhDs spending months manually tracing neural pathways. Now, Silico aims to make these insights accessible to any developer.

From Auditing to Design: How Silico Changes the Game

One of Silico’s most innovative features is its ability to intervene during model development, not just after. Traditional AI debugging happens post-training: you deploy a model, notice it’s generating harmful content or factual errors, and then try to patch it with reinforcement learning or fine-tuning. But by that point, the problematic behaviors are often deeply embedded in the model’s architecture.

Silico flips this model on its head. By analyzing the model during training, it can identify emerging circuits responsible for unwanted behaviors—like generating misinformation or exhibiting bias—and suggest adjustments before those patterns solidify. For example, Goodfire has already used its techniques to reduce hallucinations in LLMs by identifying and suppressing neurons that tend to generate fabricated facts.

This proactive approach could revolutionize how AI is built. Instead of training a model and hoping it behaves well, developers could use Silico to “steer” the model toward desired outcomes from the start. Want a model that’s more factual? Silico can highlight which data sources or training signals reinforce truthfulness. Need to reduce toxicity? It can pinpoint the neural pathways that activate during offensive language generation.

The Rise of AI Agents in Interpretability

A key breakthrough enabling Silico’s automation is the use of AI agents to perform interpretability tasks. Previously, mapping neural circuits required human researchers to manually inspect thousands of activations, a process akin to finding a needle in a haystack the size of a planet.

Now, with advances in agent-based AI, Silico can deploy autonomous software agents to scan model internals, correlate neuron activity with specific behaviors, and generate hypotheses about how circuits function. These agents can work around the clock, analyzing millions of data points and surfacing insights in real time.

“Agents are now strong enough to do a lot of the interpretability work that we were doing using humans,” Ho explains. “That was kind of the gap that needed to be bridged before this was actually a viable platform that customers could use themselves.”

This shift mirrors the broader trend of AI automating AI development. Just as GitHub Copilot helps programmers write code, Silico aims to help AI engineers understand and control their models. The result is a faster, more scalable path to reliable AI.

Real-World Impact: Safer, More Reliable AI Systems

The implications of mechanistic interpretability extend far beyond academic curiosity. In high-stakes domains like healthcare, finance, and law, AI systems must be not only accurate but also transparent and accountable. A model that recommends a medical treatment or approves a loan must be able to explain its reasoning—especially when things go wrong.

Silico could play a crucial role in building such systems. For instance, a hospital deploying an AI diagnostic tool could use Silico to verify that the model isn’t relying on spurious correlations (like a patient’s zip code) to make predictions. Similarly, a financial institution could audit its credit-scoring model to ensure it’s not discriminating based on race or gender.

Beyond compliance, interpretability also improves performance. By understanding why a model fails, developers can make targeted improvements instead of relying on brute-force scaling. This could lead to smaller, more efficient models that are easier to deploy and maintain.

The Road Ahead: Challenges and Opportunities

Despite its promise, mechanistic interpretability is still in its infancy. Mapping neural circuits in large models remains computationally expensive, and many behaviors emerge from complex interactions that are hard to isolate. Moreover, as models grow more sophisticated, their internal logic may become so intricate that full interpretability is impossible.

Still, Goodfire’s work represents a critical step toward more responsible AI. By making interpretability tools accessible to a broader range of developers, the company is helping to democratize AI safety and reliability.

Looking ahead, Ho envisions a future where AI development is as predictable as building a bridge or writing software. “We’re not just trying to fix models,” he says. “We’re trying to change how people think about building them.”

As AI becomes more embedded in society, the need for transparency and control will only grow. Tools like Silico may not unlock the full mystery of artificial intelligence—but they’re bringing us closer to a future where we can trust, understand, and ultimately improve the machines we build.

This article was curated from This startup’s new mechanistic interpretability tool lets you debug LLMs via MIT Technology Review