Making AI operational in constrained public sector environments

The Quiet Revolution: How Small Language Models Are Quietly Powering AI in Government

In the race to adopt artificial intelligence, governments are moving at a pace that would make a tortoise blush. While tech giants and startups deploy massive generative AI models with fanfare, public sector institutions are quietly building a different kind of AI future—one rooted in control, security, and operational reliability. The secret weapon? Small language models (SLMs), purpose-built for the unique realities of government work. These compact, efficient AI systems are proving that bigger isn’t always better—especially when national security, citizen privacy, and mission-critical operations are on the line.

Unlike the private sector, which often operates with a “move fast and break things” ethos, government agencies must navigate a labyrinth of legal, ethical, and technical constraints. From classified data handling to legacy IT systems, the public sector’s AI journey is less about innovation for innovation’s sake and more about doing more with less—without compromising safety or sovereignty. In this high-stakes environment, small language models are emerging as the pragmatic backbone of a new era of public service AI.

The Public Sector’s AI Dilemma: Innovation vs. Integrity

Public sector organizations are under increasing pressure to modernize. Citizens expect digital services on par with private platforms, and policymakers demand data-driven decision-making. Yet, the path to AI adoption is fraught with obstacles that don’t exist—or are far less severe—in the corporate world. A Capgemini study revealed that 79 percent of public sector executives globally are wary about AI’s data security, a statistic that underscores the deep-seated caution within government institutions.

This wariness isn’t paranoia—it’s prudence. Government data often includes personally identifiable information (PII), national security intelligence, healthcare records, and financial data, all of which are subject to strict legal protections. Sending such sensitive information to third-party cloud platforms or open AI models poses unacceptable risks. As Han Xiao, vice president of AI at Elastic, explains, “Government agencies must be very restricted about what kind of data they send to the network. This sets a lot of boundaries on how they think about and manage their data.”

Unlike private companies, which can often absorb the cost of a data breach or pivot quickly after a misstep, government agencies face lasting reputational damage, legal consequences, and public distrust. This makes risk aversion not just a preference but a necessity. The result? Many promising AI pilots stall at the prototype stage, unable to scale due to unresolved governance and security concerns.

Why Big Models Don’t Fit in Government Boots

The AI boom has been dominated by large language models (LLMs) like GPT-4 and Llama 3, which require massive computational power, continuous cloud connectivity, and vast datasets. These assumptions work well for Silicon Valley startups but crumble in government environments. Consider the typical private-sector AI deployment: data flows freely to centralized cloud servers, models are updated in real time, and performance is measured in user engagement or profit.

Government operations, by contrast, often unfold in disconnected or low-bandwidth environments—think rural health clinics, military field operations, or disaster response zones. In these settings, relying on cloud-based LLMs is not just impractical; it’s dangerous. A dropped connection could mean a failed diagnosis or a delayed emergency response.

Moreover, many government systems run on legacy infrastructure that can’t support the GPU-intensive demands of large models. Procuring high-end graphics processing units (GPUs) is not only expensive but also logistically challenging due to export controls and supply chain bottlenecks. For agencies already stretched thin, the cost and complexity of deploying LLMs are prohibitive.

Small language models, by contrast, are designed to run efficiently on modest hardware. They require less memory, consume less power, and can operate offline—making them ideal for environments where connectivity is unreliable or nonexistent. Their smaller size also means faster inference times, which is critical for real-time applications like fraud detection or emergency dispatch.

The Rise of the Small Language Model: Precision Over Power

Small language models (SLMs) are not just scaled-down versions of their larger cousins. They are purpose-built for specific tasks, trained on curated datasets, and optimized for low-latency, high-reliability performance. While an LLM might have hundreds of billions of parameters, an SLM might operate with just a few hundred million—or even fewer—delivering focused intelligence without the bloat.

This specialization is a game-changer for government use cases. For example, an SLM trained on legal documents can assist public defenders in drafting motions without exposing case details to external servers. Another SLM, fine-tuned on public health data, can help epidemiologists track disease outbreaks in real time, even in remote regions with limited internet access.

SLMs also offer greater transparency. Because they are smaller and more interpretable, auditors and regulators can more easily verify how decisions are made—a crucial requirement in public accountability. This “glass box” approach contrasts sharply with the “black box” nature of many LLMs, where outputs can be unpredictable or unexplainable.

Operational Continuity: The Unseen Priority

One of the most underestimated challenges in public sector AI is ensuring continuity of operations. In the private world, a system outage might mean lost revenue or customer frustration. In government, it can mean lives lost or services denied. “Many people undervalue the operating challenge of AI,” says Xiao. “The public sector needs AI to perform reliably on all kinds of data, and then to be able to grow without breaking.”

An Elastic survey of public sector leaders found that 65 percent struggle to use data continuously in real time and at scale. This isn’t just a technical issue—it’s a systemic one. Legacy systems, fragmented data silos, and outdated workflows make it difficult to integrate AI without disrupting critical functions.

SLMs address this by being inherently more stable and predictable. Because they are trained on domain-specific data and designed for narrow tasks, they are less prone to hallucinations or erratic behavior. They can be deployed incrementally, tested in controlled environments, and scaled only after proving their reliability.

Consider the example of a state transportation department using an SLM to analyze traffic patterns and optimize signal timing. The model runs on local servers, processes anonymized sensor data, and adjusts signals in real time—without ever sending data to the cloud. If the internet goes down, the system continues to function. This kind of resilience is non-negotiable in public infrastructure.

The Edge Advantage: AI Where It’s Needed Most

One of the most powerful applications of SLMs is at the “edge”—the point where data is generated and decisions must be made instantly. In military operations, for instance, soldiers in the field can’t wait for a cloud-based model to respond. They need AI that works on a rugged tablet or handheld device, analyzing satellite imagery or translating local dialects in real time.

SLMs make this possible. By processing data locally, they eliminate latency and reduce bandwidth demands. They also enhance security, as sensitive information never leaves the device. This edge computing approach is already being tested by NATO forces and disaster response teams, where every second counts.

Building Trust Through Transparency and Control

Trust is the currency of government. Without it, even the most advanced AI will fail. Citizens and lawmakers alike demand to know how decisions are made, especially when those decisions affect their rights, safety, or resources. SLMs, with their smaller size and focused training, offer a path to greater transparency.

Unlike LLMs, which often rely on opaque training data and complex neural architectures, SLMs can be designed with explainability in mind. Developers can trace how inputs lead to outputs, audit training datasets for bias, and provide clear documentation for regulators. This is essential for compliance with emerging AI regulations, such as the EU’s AI Act or the U.S. Executive Order on Safe, Secure, and Trustworthy AI.

Moreover, SLMs empower agencies to retain full control over their data and models. They can be hosted on-premises or in government-owned cloud environments, ensuring that no third party has access to sensitive information. This sovereignty is particularly important for national security and law enforcement agencies, where data leakage could have catastrophic consequences.

The Road Ahead: A New Model for Public AI

The future of AI in the public sector won’t be defined by the size of the model, but by its fit for purpose. As agencies continue to grapple with budget constraints, security mandates, and operational complexity, small language models offer a pragmatic and scalable solution. They may not generate flashy headlines, but they are quietly transforming how governments serve their citizens.

From streamlining benefits processing to enhancing national defense, SLMs are proving that AI doesn’t need to be big to be powerful. What it needs is precision, reliability, and trust—qualities that are not just desirable in government, but essential.

As Han Xiao aptly puts it, “The goal isn’t to build the smartest AI. It’s to build the right AI for the mission.” In the constrained, high-stakes world of public service, that distinction has never mattered more.

This article was curated from Making AI operational in constrained public sector environments via MIT Technology Review