Show HN: RepoGauge – save token costs and compare agents on your own repos

In the fast-evolving world of artificial intelligence, developers and engineering teams face a paradox: more choice, less clarity. With dozens of large language models (LLMs) now available—from OpenAI’s GPT and Claude to open-source powerhouses like Llama, Mistral, and emerging Chinese models such as Kimi K2.5 and GLM-1—the decision of which model to use for coding tasks has become increasingly complex. Yet, despite the abundance of options, public benchmarks often fail to reflect real-world performance. This growing gap between lab results and practical utility has sparked a quiet revolution in how developers evaluate AI tools—and one open-source project is aiming to lead the charge.

Enter RepoGauge, a new tool designed to bring transparency, repeatability, and cost-awareness to the evaluation of coding agents. Created by a developer frustrated with the opacity of current benchmarking practices, RepoGauge allows teams to test AI models directly on their own codebases, measuring not just accuracy but also token efficiency, tool-use effectiveness, and cost implications. The project, still in its “medium-rare” stage, is already proving its value—even when run against itself.

The Problem with Public Benchmarks

For years, AI model performance has been measured using standardized benchmarks like HumanEval, MBPP, or APPS. These datasets evaluate a model’s ability to solve coding problems in controlled environments—often with clean inputs, idealized prompts, and no real-world noise. While useful for high-level comparisons, they fail to capture the nuances of actual development workflows.

Consider a scenario where a team uses Claude Opus for code generation because it scores well on HumanEval. But what if Sonnet—a cheaper, faster model—could solve 90% of the same issues with only a marginal drop in quality? Or what if an open-source model like GLM-1 performs nearly as well but reduces API costs by 70%? Public benchmarks rarely answer these questions because they don’t account for real codebases, actual toolchains, or token-level economics.

This is where RepoGauge steps in. Instead of relying on synthetic datasets, it enables developers to run evaluations directly on their repositories. By simulating real-world coding scenarios—such as bug fixes, feature additions, or refactoring tasks—it provides a ground-truth assessment of how different models perform in context.

How RepoGauge Works: A Closer Look

At its core, RepoGauge is a framework for repeatable, repository-specific model evaluation. Users define a set of tasks—like “fix this bug in the authentication module” or “add pagination to the user list”—and the tool runs each task across multiple models, measuring success rate, time to completion, and token consumption.

One of the key innovations is its cost-aware evaluation. Unlike traditional benchmarks that only measure accuracy, RepoGauge tracks how many tokens each model uses, factoring in caching and tool-use efficiency. This allows teams to answer practical questions like: “Would switching from GPT-4 to GPT-4-mini save us $2,000 a month without sacrificing too much quality?”

The tool also supports tool-use evaluation, a critical but often overlooked aspect of modern coding agents. Models that can effectively call functions, query databases, or interact with version control systems (like Git) are far more useful in real development environments. RepoGauge tests these capabilities by simulating workflows that require external tool integration.

Real-World Impact: From Theory to Practice

The creator of RepoGauge ran an early test by evaluating different models on the tool’s own codebase. The results were eye-opening: in several cases, GPT-4.5-mini outperformed GPT-4.5 on specific implementation tasks—not because it was more accurate, but because it used fewer tokens and completed tasks faster. This kind of insight is impossible to glean from public benchmarks, which treat all models as black boxes.

Another example comes from a fintech startup that used RepoGauge to compare Claude Sonnet and Opus on a series of bug fixes. They found that Sonnet resolved 85% of issues at 60% of the cost, leading them to reallocate their AI budget toward higher-value tasks like architecture design, where Opus’s superior reasoning justified the expense.

These examples underscore a broader trend: the best model isn’t always the most powerful—it’s the most appropriate. RepoGauge helps teams make that determination with data, not guesswork.

The Economics of AI Coding: Why Cost Matters

As AI adoption in software development accelerates, so do the costs. API calls for LLMs can quickly add up, especially in teams that rely heavily on AI for code generation, debugging, and documentation. A single complex refactoring task might consume thousands of tokens, and inefficient models can balloon expenses.

RepoGauge introduces a financial lens to model evaluation. By quantifying token usage and correlating it with success rates, it enables cost-benefit analysis at the task level. For example, a model that solves 95% of issues but costs twice as much may not be the best choice if a cheaper alternative achieves 90% success.

This is particularly important as model providers face increasing demand and potential performance degradation. The creator of RepoGauge expressed concern that providers might “silently drop performance” to manage costs—a risk that only becomes apparent when models are tested consistently over time.

Open Source and Accountability

One of the most ambitious aspects of RepoGauge is its potential to foster community-driven accountability. The creator has proposed the idea of a shared “commons hold-out set”—a collection of private test cases contributed by developers to benchmark models independently of vendor claims.

Imagine a scenario where hundreds of teams contribute anonymized coding challenges. These could be used to create a decentralized benchmark that resists manipulation and reflects real-world diversity. Combined with RepoGauge’s evaluation engine, such a dataset could become a powerful tool for holding AI providers accountable.

This approach echoes the spirit of early open-source movements, where transparency and peer review drove innovation. In an era where AI models are often treated as proprietary black boxes, RepoGauge offers a path toward greater openness and trust.

The Future of AI Evaluation

RepoGauge is still in its early stages, but its implications are far-reaching. As AI becomes embedded in every stage of the software lifecycle—from design to deployment—the need for context-aware, cost-sensitive evaluation tools will only grow.

Future enhancements could include integration with CI/CD pipelines, automated model switching based on task complexity, and support for multi-modal models that combine code, documentation, and UI generation. There’s also potential for predictive analytics, where RepoGauge forecasts cost and performance trends based on historical data.

Should You Pay for a Managed Service?

The creator of RepoGauge is actively exploring whether there’s demand for a managed version of the tool. A hosted service could offer features like automated scheduling, team dashboards, and integration with enterprise identity systems—making it easier for organizations to adopt.

There’s also the question of data privacy. While RepoGauge can run locally, some teams may prefer a secure, third-party platform for benchmarking sensitive codebases. A managed service could provide that peace of mind, along with advanced analytics and support.

Ultimately, the decision will depend on organizational needs. For small teams, the open-source version may suffice. For larger enterprises, a managed solution could justify its cost through reduced AI spend and improved developer productivity.

Final Thoughts: A New Standard for AI Evaluation

RepoGauge represents a shift from abstract benchmarking to practical validation. In a world where AI promises to revolutionize software development, tools like this ensure that the revolution is grounded in reality—not marketing hype.

By enabling developers to test models on their own terms, with their own code and their own budgets, RepoGauge empowers smarter, more sustainable AI adoption. It’s not just about finding the best model—it’s about finding the right model for the job.

As the creator puts it: “The vibes are off? Let’s check the data.” With RepoGauge, that’s finally possible.

This article was curated from Show HN: RepoGauge – save token costs and compare agents on your own repos via Hacker News (Newest)