Disclaimer: like many blog posts about AI, this one is likely to be massively out of date if read well after its publication, but may nonetheless be interesting as a record of capabilities and expectations at the given time point.

Around the start of this year, I wrote a blog post about using AI to write high-quality code. My key takeaway was that, for applications involving complex low-level details that need to be handled correctly, AI-assisted human-in-the-loop workflows are much faster than writing code by hand—and, in particular, that they actually work. This state of affairs was clear to some people at the time, but not to everyone. Today, it is clear to essentially anyone who is paying attention, and who has avoided falling into the trap of motivated reasoning.

Since then, AI capabilities has not stopped improving. A few weeks ago, an agent swarm operated by OpenAI autonomously solved Navier–Stokes—with plenty of controversy of no relevance to the topics here. If an AI system can do that, is human input needed to build a similar codebase to what I wrote about less than a year ago? To find out, I decided to put together an experiment.

Code for this blog post is available on GitHub.

PDF Compression From-scratch

I am somewhat embarrassed to start this writeup by admitting that my downloads folder has been a mess for years. In particular, it has accumulated over a thousand academic papers over the last half-decade—in spite of the fact that I use reference management software to keep track of what I’ve read. Given the availability of coding agents, sorting those papers and importing them into the database should not be hard. And yet, before this blog post, I hadn’t done it.

Why not? Of all things, the bottleneck has been PDF compression. You see, some of those papers have huge graphics and very poorly optimized PDFs—the kind that not only use up gigabytes of storage, but sometimes even crash your viewer if you scroll to the wrong page. I keep my papers in the cloud and like to have them downloaded on my phone, in case I’m working somewhere with limited cellular data. Without compression, I would’ve run out of space.

At the same time, open-source PDF compression tools leave much to be desired. GhostScript and related tools deal mostly with images, but sometimes PDFs have huge vector graphics, huge fonts, huge metadata, and are often poorly optimized for other reasons. The best tool I know is ILovePDF.com—a commercial tool with a web interface. It offers excellent compression and compatibility, but does not offer the kind of command-line interface that would be convenient for batch compressing my downloads.

In the old days of software, I would have had to stop there. PDF is not a simple format, processing it requires fragile-glass codeⁱ as much as CUDA kernel do. Peter Steinberger of OpenClaw fame previously built a whole company in this domain. If I wanted my own PDF compression app, my only realistic option beyond open source was to pay for software built by someone who dedicated a lot of time to getting the details right. What about today? PDF compression is hard—but, surely, solving Navier–Stokes is much harder. Why can’t AI just do it?

Evaluation-driven Development

To find out, my first step was a way to write down what kind of tool I actually want—what should be built, what is the interface, what are the features. To get started, I opened Claude Code, and told the model that I was very curious why ILovePDF is so much better than GhostScript in practice. I gave it a few example PDFs to examine in order to find out how.

From this, Claude came up with a rather complex list of about twenty individual compression techniques it thought were making a difference. I then asked Claude how hard it would be to implement all of them in Rust, including which existing libraries we could use for various low-level functionality. Claude wrote a Markdown table with this information. Together with a few small details about the desired command-line interface, this table became the start of a spec. You can view it here.

A software specification tells us what to build—but, by its nature, a spec is a summary document. Actually building software requires many low-level decisions on how to implement the spec. I’m not a PDF format expert, and had neither the knowledge nor interest in participating in these decisions. At the same time, I needed a way to know that they were done competently. So, I asked Claude to look at open source libraries, and assemble a comprehensive set of PDFs to form a test suite—and it did, with nearly 5000 files total.

This development pattern has a name: evaluation-driven development. The idea is that we specify most of the necessary information about what our software should do by providing a set of examples of correct behavior. This includes, ideally, evaluations that catch important edge cases. In our case, each example is a PDF file. Using the evaluation suite as a source of truth, we then ask an AI agent to build our software for us.

On the morning of September 15th, after finalizing the initial spec, assembling the aforementioned eval suite of about 5000 PDFs, and completing a few other small miscellaneous tasks,ⁱⁱ I asked Claude to attempt to actually build the whole tool. I then left my laptop alone in my hotel room—at the time, I was attending a small conference—and Claude Code run on its own, while I went hiking with other participants.

Around the hike’s halfway mark, I learned from my phone that Claude Fable 5.1 completed its task. At completion time, it had used up half of my weekly subscription limits. When I returned from the hike, I discovered that almost all of the featured I asked for were implemented. This includes a few that involved significant custom code not covered by libraries.

The codebase checks correctness in two ways. First, it uses a library to render the PDFs before and after, and checks the resulting images are sufficiently-similar at the pixel level. This catches most data-loss errors. Second, it passes the PDFs through a different parser compared to the one it uses for making changes, to check format validity. Already for the first version, completed during the hike, this gave me confidence the code worked as intended.

It took only a few additional prompts, made in-between talks during the remainder of the conference, to add the remaining features. The code successfully compressed almost all of the thousand PDFs it was intended for on the first try. A small set of edge cases errored, but were fixed after a prompt or two. A week later, I took a fresh look at the code—and, a few prompts later, reorganized it around a reasonable API that fits Rust ecosystem expectations. The result of this experiment is compress-pdf, available on GitHub and crates.io.

Some Implications

In developing this code, I used a completely different style than I’ve ever worked with pre-AI. I find it astonishing how little of my time or input was needed to complete it, in spite of the complexity of engineering or difficulty of correctness. Most of my actual work involved interactively prompting the model to write a reasonable spec—but, to some approximation, that’s it. I find it doubly astonishing how few tokens were used, in spite of discourse I hear online.

This style of development works. A lot of people, especially those close to the AI ecosystem,¹ have adopted it already—and, I predict, far more people are going to adopt it soon. The time-quality tradeoff is sufficiently different from pre-AI software development that I predict it will trigger a lot of downstream consequences—most immediately, on what kind of software companies are financially viable, and how many people are needed to run them.

A week later, I’ve found that this experiment has inspired broader changes in how I work. After completion, the very next thing I did was asked GPT-6 Astra—which I had heard was particularly strong at math—to work on a certain conjecture in multi-armed bandits that I am personally very interested in and have been using to evaluate LLM capabilities. No previous model has, in my opinion, successfully made any progress in resolving it.

Astra thought for about an hour, and produced a counterexample. That’s fine—various models, mostly Opus variants, have also claimed to produce proofs—but all were clearly incorrect. I scanned the counterexample’s proof and, in five minutes, determined the approach seemed sane. I could have then checked the logic carefully but, after my experience, was inspired to do something different: I asked Astra to formalize it in Lean. Half a day later, Astra succeeded.

By itself, even this is not enough to instill full confidence: it’s possible that a model correctly formalizes the wrong statement. To make a long story short, I am confident that’s not what happened here. Thus, I consider Astra to be the first large language model to successfully solve the simplest form of the conjecture I’ve been using to privately evaluate AI capabilities—one that is sufficiently hard I would’ve given it 70%-30% proof-counterexample chances.

To me, these outcomes are strong evidence of artificial general intelligence. If you don’t find them sufficient, what , why are your definitions so specific?

Sources of Truth

In thinking about this experiment, a key concept that has helped me make sense of things is that of sources of truth. These allow us to say on what basis we conclude that a codebase is implemented correctly.

The code itself is one source of truth: we can learn what it does by reading its lines. Unit tests are another source of truth: they can tell us whether the code does what we think it does, in a narrow context. Formal verification is a third, previously rarely-used, source. Most approaches to testing can be thought of similarly. In some sense, one can even say that trusting other developers to do a good job is a source of truth about correctness.

One does not need to be tied to a particular source of truth to gain confidence in a codebase doing what it is supposed to do. In this exercise, a combination of the spec, system design and API surface, and evaluation suite—and, of course, my trust in Claude to not do anything outright evilⁱⁱⁱ—form the source of truth. The evaluation suite, in particular, has many edge cases—including ones collected over many years by PDF libraries. I trust it. If you don’t, why not?

Stepping back somewhat, I believe that, as a result of AI, designing sources of truth becoming a central focus of software engineering. This is almost by definition: given the possibility of errors and their consequences, we need ways of verifying correctness—otherwise, we will quickly lose control over what we build. I am excited to see the creative ways in which people start doing figuring out how to understand what the systems we build actually do.

Conclusion

In this blog post, I described how I built a PDF compression command-line tool using an AI-assisted workflow with minimal input. The core concepts that helped me think about this process were that of evaluation-driven development, and that of sources of truth about what a codebase does.

There is far more to say about the implications than the narrow thoughts I’ve written down here, which are inspired as much by what I was doing day-to-day as by changes I think will follow as a result of what is now possible. Figuring those out is incredibly important, and I plan on writing more on these topics soon.

Footnotes