Verifying LLM output
for academic economists

Verification is becoming a bottleneck for empirical research. AI agents are fast, cheap, and competent at producing code, including in languages that you may not know (I don't really know how to make a website, but Claude does). While I have seen plenty of concerns over verification, I have not seen a practical guide that collects ways of actually verifying LLM output.

This guide is meant to address this gap. These checks are collected from my own experience and from others, including Geoffrey Litt on understanding as the new bottleneck and Paul Goldsmith-Pinkham on integration and collaboration in AI research work. If you have other checks, please share them — this is meant to be a living document.

First Principle

You have to read the code — or some version of it

The code is the analysis. You have to do some version of checking that the code does what you actually want it to do. Now, you can do that in many ways, and some are going to be more efficient than others. For instance, if you use Git you can choose to read only the parts that change (that is easy to get out of Git). You can also have your favorite agent present the changes to you along with an explanation.

I'm figuring out how to read code updates from Claude myself as I go along, and I don't know if I have the right approach yet. But what is clear is that you are responsible for the code once it produces results that are in the paper, and so you have to do some work yourself.

Something to keep in mind: you are not really looking for the normal errors that would break a code or cause a crash. Claude and agents are very good at iterating and figuring out what breaks code, and they can work autonomously to fix those errors. So you will get a table with output, and it might look plausible. But it might be wrong because the agent took a shortcut. The errors you are looking for are the subtle ones that don't crash but lead to incorrect conclusions.

Prerequisite

Use git to make verification tractable

Git is a very useful tool for verification, because it allows an agent to evaluate and verify the code that changed. Git allows you to see what changed using git diff, which also means that the agents can easily see what changed. I recommend watching and reading Paul Goldsmith-Pinkham on integration and collaboration in AI research work.

Using git matters more for agents than for us people. If you ask the agent to review the code, it will look over all the code and it will spread its attention over too many things. This invites both generic answers, a lack of focus, and will cost you more tokens. If the agent instead examines a diff, it will know exactly what is new, and can compare the change against the code you had already checked.

Check One

01Launch a fresh agent to attack the diff

Launching a fresh agent or a fresh chat window to evaluate the code is an easy but important first check. The reason you want to use a fresh agent/chat is that the agent that wrote the code has too much context. Because it wrote the code, it has all the conversation in mind, and it is not an independent arbiter of what is right and wrong. It's very easy to get around that problem by launching a fresh agent: either you ask the current agent to launch a subagent (Use a subagent to review the code…), or you start a new chat window with a reference to the code you want checked. A fresh agent has no stake in the code, and will be more likely to find problems.

One caveat on subagents: the parent writes their instructions and summarises the report back to you, so you read a filtered account. A new session you brief yourself is the more isolated option.

Check Two

02Use a different model

This one is very similar to the above, in that you have a fresh agent look at the code. But sometimes there is a value in having a different model entirely look at the code. Two instances of the same model share training data and habits of thought, so they tend to make the same mistakes.

Running the diff past a model from a different developer breaks that correlation, on the same logic as sending a paper to two referees from different subfields rather than to two students of the same advisor. Setting up both tools is covered separately in the Claude Code guide and the Codex guide.

Check Three

03Make the agent quiz you

The checks so far verify the code using LLMs. But you are also supposed to be in the loop, which is easy to skip. Using a quiz is a nice way to make sure that you understand what is going on. Geoffrey Litt's explain-diff skill is a neat solution. The agent writes a self-contained HTML page explaining a change — background, the core intuition with toy examples and diagrams, a walkthrough of the code — ending in five multiple-choice questions with immediate feedback.

Here is the instructions from the skill:

Quiz: Come up with five questions that test the reader's
knowledge of this PR. This should be medium difficulty,
difficult enough that you actually need to understand 
the substance of the PR to answer them, but not gotchas.
The goal is to help the reader make 
sure that they've actually understood. These should be 
presented as interactive multiple-choice questions, and when
the user clicks, it tells them whether they were correct 
and gives feedback.

Adapted for an empirical paper rather than a software change, the skill looks like this. Save it as ~/.claude/skills/quiz-me/SKILL.md and it becomes /quiz-me.

---
name: quiz-me
description: Explain a change to the analysis and quiz me on whether
  I understood it. Use when I have asked for a change to cleaning or
  estimation code and need to verify my own understanding.
disable-model-invocation: true
---

Work only from the code and the diff. If this conversation contains
an earlier account of the change, ignore it and read the files.

Produce a single self-contained HTML file explaining the specified
change, ending in a quiz. Save it outside the repository with a
filename starting with today's date in YYYY-MM-DD- format, then
tell me the path.

Sections:

1. What this code did before. Explore the surrounding scripts, do
   not rely on the diff alone.
2. What changed and why, in plain language. No code in this section.
3. Consequences for the results. Which sample, which coefficients,
   which tables are affected, and in which direction. Say so
   explicitly if the answer is none.
4. Walkthrough of the changed code, grouped by purpose rather than
   by file.
5. Quiz. Five multiple-choice questions, medium difficulty — hard
   enough that answering requires understanding the substance, not
   gotchas. At least two must be about empirical consequences
   rather than syntax: which observations enter the sample, what
   the coefficient now identifies, what would change if an
   assumption failed. Distractors must be plausible
   misunderstandings, and must match the correct answer in length,
   grammar, specificity, and confidence. Randomise the position of
   the correct answer independently for each question. On click,
   report whether the answer was correct and explain why, including
   why each wrong option is wrong. For each question, cite the file
   and line the answer rests on.

Style: clear prose, concrete toy examples with made-up numbers, and
simple HTML diagrams rather than ASCII art. Code in `<pre>` tags.
Embed all CSS and JavaScript so the file works offline.

Run it in a new session rather than the one that made the change. Starting fresh forces the agent to read the files instead of recalling the conversation, which is why the skill opens by telling it to.

Check Four

04Reimplement the analysis in another language

StataRPython This is a check that Scott Cunningham has advocated for (at least he's the person I think about when this check comes up). It is the most expensive check, but it is also the strongest: you ask for code in different programming languages to see if you get the same output.

This is a strong test because it involves the whole chain. If the errors across languages are independent, then you should be able to spot them by looking at the differences in output. An independent implementation across programming languages catches the case where the do-file faithfully implements something other than what the paper says it does. Use a different model for it as well, for the reason in check 02. Given how good agents are becoming at writing code, this test is getting cheaper to run over time as well.

Isolating the second implementation

Now, this check can also happen organically as you work with agents. Claude is very prone to using Python for generating intermediate results in my chats, for example. If you open a separate chat and ask for a Stata pipeline and the numbers match, you kind of get this check for free. As with many other checks, verification is not a one-time thing but rather a continuous process.

But it is also worth thinking about how to implement the check independently. To do that, you need to think about how LLMs work, and make sure that you isolate the model from the previous context. If you ask "Ignore what you know about 02_analysis.do", you essentially ask the model to un-condition on what is already in its context, which is not something it can do. The context is always loaded. You have to enforce independence by removing the context that could contaminate the results. You can do that by creating a new folder with a fresh session. Then add a file with the specification or a description of what you want to implement and a description of the data.

You can probably figure out multiple ways to do this. One way: copy the specification description out of the paper into spec.md and delete the results from it. Also write (or have your agent write) a file that describes the data and the variables (README.md). Then ask your favorite model some version of:

Read verify/spec.md, which describes a specification, and
data/README.md, which describes the raw data. From those two
documents only, write an R script that estimates the specification
starting from the raw files in data/raw/. Save it as
verify/analysis_check.R.

Where the description is ambiguous, do not guess. Stop and list
the ambiguities and I will resolve them.

Do not run it. I will run it myself and tell you what it returns.

Running it yourself is important (and not just for saving tokens). An agent that runs its own script and sees an implausible number will debug towards plausibility, and plausibility is exactly what you are trying to test. Similarly, the instruction to stop at ambiguities may help you resolve some ambiguities yourself.

Check Five and Onwards

Checks that cost almost nothing

It is also worth creating some habits. Here are some examples.

Verify the data, not only the code

Correct-looking code operating on the wrong sample produces wrong results. Make sure you know what the data looks like. A sample-construction table is a simple way to do this. Also make sure that you check that merges work as expected, and take a look at the distribution of constructed variables.

Compare against an external benchmark

We all do this when figuring out whether the results "make sense", and so we should keep doing it also for LLM output. This is almost certainly a part of your normal workflow, so it may not be a huge extra cost. This is an external validation, which is a nice way to catch errors that are not caught by internal checks from above.

Check the numbers in the prose against the numbers in the tables

Coefficients get quoted in the abstract and the introduction, the specification then changes, and you don't update the writing. This kind of consistency check is also included in my review-paper skill (for this skill and more, see my collection of AI skills). This is a tedious check for a human and a trivial check for an agent.

Verify your references

Models still invent citations. There are now some automated checks for references, like Reviewer3, and probably others.

Before the Checks

The cheapest verification happens while the code is being written

All the above checks are after-the-fact checks. Each assumes code exists and asks how to establish that it is right. But the most reliable way to understand an analysis is to have been present while it was built.

Compare two sessions. In the first, you describe the whole pipeline in a paragraph, the agent works for six minutes, and you are handed 400 lines across five files. Verifying that means reverse-engineering someone else's reasoning from its output, which is the position a referee is in, except that you are the author. In the second, you agree on a plan, the agent asks three questions you had not thought about, then writes one cleaning step, runs it, and shows you the counts. Both produce the same script, but in the second you already know what it does, and the checks below become confirmation rather than discovery.

The second is also not slower in the end. Time saved by one long prompt is borrowed against the debugging session where you discover, three tables later, that the deflator was applied twice. The exception is mechanical work with a visible output, where you can see whether the result is right and none of this is needed.

A working shape for the loop, using a panel construction task as the example.

1. Ask for a plan, not code. A plan is a page of prose you can read in a minute, and disagreeing with it costs nothing. Disagreeing with a finished implementation costs a rewrite. Most tools have an explicit mode for this — otherwise say so in the prompt.

   Before writing any code, describe how you would build the
   firm-year panel from the raw files: which files, in which
   order, what the key is at each stage, and where observations
   could be lost. Number the steps so I can respond to them
   individually. Do not write code yet.

2. Make it ask you questions. An agent that has to ask whether firms changing industry mid-panel keep their original classification has surfaced a decision that would otherwise have been settled by whichever default it picked, and that you would have discovered at the earliest when a coauthor asked.

   Before you start, ask me every question you need answered
   to do this correctly. I would rather answer ten questions
   now than discover your assumptions later. Include the ones
   where you could guess a reasonable default — I want to know
   which defaults you would have chosen.

3. Work in steps small enough to check, and run each one. Code that has not been executed is a hypothesis. Ask for one step, have it run, look at the output, then continue. The counts after each step are the verification, and they arrive while the decision is still fresh enough to discuss.

   Implement step 1 only. Run it, then report the number of
   observations and unique firms, and show me the first ten
   rows. Stop there and wait for me before starting step 2.

4. Ask why, not what. A summary of what the code does restates the code. The useful question is about the decisions, the places where something else was possible, because that is where your judgement is needed. Note that the second half of this prompt asks for something the agent has to run rather than assert.

   List the three decisions in what you just wrote where a
   reasonable person could have done something different, and
   what you chose in each case. Then re-run the specification
   under each alternative and give me the coefficients side by
   side. Do not tell me what you expect the alternative to do —
   run it.