Show HN: Neural window manager, neural network moving windows from mouse actions

Imagine a world where your computer’s interface doesn’t rely on lines of code to decide how windows move—instead, it learns how to behave by watching you. No event handlers, no coordinate tracking, no traditional programming logic. Just a neural network that watches your mouse, sees the screen, and predicts what should happen next. This isn’t science fiction—it’s a real, working experiment that flips decades of user interface design on its head.

At its core, this breakthrough explores a radical idea: can software be generated, not programmed? Inspired by advances in AI world models—systems that simulate environments by predicting future states—a developer set out to build a neural window manager that generates screen behavior pixel by pixel. The result is a fascinating blend of machine learning, human-computer interaction, and computational minimalism that challenges how we think about software architecture.

The Radical Idea: From Code to Prediction

For over half a century, graphical user interfaces (GUIs) have operated on a simple but rigid principle: code defines behavior. When you click and drag a window, the operating system tracks your cursor’s coordinates, calculates the new position, and redraws the window accordingly. It’s a deterministic process built on logic, state, and explicit instructions.

But what if we could skip the code entirely? What if, instead of programming rules, we trained a model to predict what the screen should look like after a mouse movement? This is the audacious premise behind the neural window manager. Instead of writing functions to handle drag events, the system learns to generate the next frame based solely on what it sees—the previous two frames and the mouse’s position and movement.

This approach draws inspiration from world models in AI, systems that simulate environments by predicting future states from sensory input. Think of it like teaching a robot to play a video game by showing it thousands of gameplay sequences—no rules, just pattern recognition. Here, the “game” is a desktop interface, and the “player” is a neural network learning to mimic window behavior.

The implications are profound. If a neural network can learn to move windows correctly, perhaps other interface behaviors—resizing, minimizing, even launching apps—could be modeled the same way. It suggests a future where software isn’t written but trained, evolving through observation rather than instruction.

Building the Experiment: Simplicity as a Design Principle

To test this idea, the developer created a minimalist simulation using Pygame, a popular Python library for building games and interactive applications. The setup was deliberately simple: a turquoise desktop background, a single gray window with a navy blue title bar, and a white cursor. Only four colors were used—not to save memory, but to reduce complexity and help the model focus on spatial patterns rather than color nuances.

A bot was programmed to randomly drag the window across the screen, simulating human-like interaction. As it moved, every frame was recorded along with the mouse’s delta—its change in position (dx, dy) and whether a click occurred. Over the course of a few minutes, 8,000 frames were captured and processed into color index matrices, a simplified representation that avoided the overhead of full RGB values.

This data became the training set for a U-Net, a type of convolutional neural network originally designed for medical image segmentation. The U-Net’s architecture is ideal for this task: an encoder compresses the input (the last two frames and mouse data), a bottleneck layer processes the information, and a decoder reconstructs the predicted next frame. Crucially, the mouse’s movement vector was projected into the bottleneck using a linear layer and concatenated with the visual data, allowing motion cues to influence every stage of reconstruction.

The model didn’t need to understand what a “window” was. It didn’t know about pixels or coordinates. It simply learned to predict: given this screen and this mouse movement, what should the next screen look like? And remarkably, it worked. When tested, the network could drag the window smoothly, stopping when the mouse was released—all without any internal state or programmed logic.

The Limits of Pixel Prediction: When Reality Distorts

Despite its success, the pixel-prediction model had a critical flaw: it couldn’t sustain long-term accuracy. After a few seconds of dragging, the window began to distort—stretching, flickering, or even fragmenting into visual noise. This wasn’t a bug in the code; it was a fundamental limitation of the approach.

Neural networks that predict pixels frame-by-frame are prone to error accumulation. Each prediction is based on the previous one, so small inaccuracies compound over time. Think of it like whispering a message down a long line of people—each whisper introduces a tiny error, and by the end, the message is unrecognizable. In this case, the “message” is the window’s position and shape.

This distortion revealed a deeper truth: pixel-level prediction is powerful but fragile. While it’s possible to generate realistic-looking transitions, maintaining consistency over time requires either vastly more data, more computational power, or a shift in strategy.

A Pivot to Primitives: From Pixels to Predictions

Rather than fight the limitations of pixel prediction, the developer made a bold pivot: abandon rendering entirely and predict interface primitives instead. Instead of generating pixels, the system would predict the actions a window should take—how much to move (dx, dy) or resize (dw, dh)—based on simple geometric inputs.

This new model used a small Multilayer Perceptron (MLP), a type of neural network well-suited for structured, low-dimensional data. The inputs were intuitive: the distance from the cursor to the window’s title bar, the distance to the resize handle, and whether the mouse was clicked. The outputs were four numbers: changes in position and size.

The key innovation was the use of two separate output heads—one for moving and one for resizing—that shared only the click signal. This architectural choice prevented the model from confusing a drag with a resize, ensuring that each action was learned independently. It’s like teaching a robot to use two tools: a hammer and a screwdriver. Even if both are in the same toolbox, they’re used for different jobs.

This approach transformed the system from a visual predictor into a motion engine. Instead of guessing pixels, it learned the logic of window behavior—just encoded in weights and biases rather than code. And because it operated on primitives, it was far more stable, efficient, and interpretable.

Running in the Browser: The Power of ONNX

One of the most impressive aspects of this project is that it runs entirely in the browser—no server, no backend, no installation. This was made possible by exporting the trained MLP to ONNX (Open Neural Network Exchange), an open format for representing machine learning models.

ONNX allows models to be trained in one framework (like PyTorch or TensorFlow) and deployed in another (like a web browser using ONNX.js). This cross-platform compatibility is crucial for real-world applications, where users expect fast, secure, and private interactions.

In the browser, the model runs on a simple HTML canvas element. Two small neural networks—one for movement, one for resizing—communicate with each other, processing mouse input and updating the window’s state in real time. The entire system is lightweight, responsive, and entirely self-contained.

This shift to browser-based execution also opens the door to democratized AI development. Anyone with a web browser can interact with, modify, or extend the model—no specialized hardware or software required. It’s a glimpse into a future where AI-powered interfaces are as accessible as websites.

Implications for the Future of Software

This experiment is more than a technical curiosity—it’s a proof of concept for a new paradigm in software design. If a neural network can learn to manage windows, what else can it learn? Could entire applications be trained instead of programmed? Could user interfaces evolve through interaction, adapting to individual behaviors over time?

We’re already seeing similar ideas in other domains. AI-generated code tools like GitHub Copilot suggest that programming itself may become a collaborative process between humans and machines. Adaptive interfaces, like those in smartphones that learn your habits, hint at systems that evolve through use.

But this project goes further: it suggests that the behavior of software doesn’t need to be explicitly defined. Instead, it can emerge from patterns in data—just like language, art, or music.

Challenges and the Road Ahead

Of course, this approach isn’t without challenges. Neural networks are black boxes—difficult to debug, verify, or trust. If a window behaves unexpectedly, how do you fix it? Unlike traditional code, where you can trace execution step by step, neural models offer little transparency.

There’s also the question of generalization. The current model works only in a controlled environment with one window and four colors. Scaling it to real desktops—with multiple windows, varying themes, and complex interactions—would require massive datasets and far more sophisticated architectures.

Still, the experiment proves a crucial point: intelligent behavior can emerge from simple learning mechanisms. It doesn’t take a supercomputer or a billion parameters to create something that feels alive. Sometimes, all it takes is a curious mind, a few thousand frames, and the courage to ask, “What if?”

As we stand on the brink of an AI-driven future, this neural window manager reminds us that the most revolutionary ideas often start not with complexity, but with a simple, bold question: Can we do this differently?

This article was curated from Show HN: Neural window manager, neural network moving windows from mouse actions via Hacker News (Newest)