While AI has shown stunning proficiency in verifiable domains of research like math and theoretical computer science, it feels harder to imagine similarly rapid progress in research areas without formal verification. The challenge seems to lie with the informal nature of empirical reasoning itself: How do we get AI to pose the right questions, run the right experiments, and then come up with the right interpretations? How does research happen when there does not and can not exist a Lean certificate for getting these steps “right”? This challenge is not due to limited AI capabilities, but rather a thread of subjectivity and human values woven into the fabric of scientific reasoning. And since more powerful LLMs will not necessarily resolve this problem, it is worth spending time sorting out what approaches can address this challenge.

Here, I will present a framework that I find useful for thinking about various approaches to AI-automated research1. The basic idea is this: We first introduce standards, which are informal guidelines or criteria against which AI output will be judged. There might be a way to formally verify whether some AI output satisfies these criteria (e.g. “this code must compile”) – in which case we might call them rules. Or the criterion might not be formally verifiable, as will happen in the majority of cases.

Then, we can divide the tasks of automated research systems into three categories:

  1. Validation: Assessing whether the standards are conducive towards achieving a desired goal
    1. Adjudication: The process of determining whether AI output meets these standards (whenever the criteria are not formally verifiable), or…
    2. Verification: The process of determining whether AI outputs are correct (whenever criteria are formally verifiable)

This terminology (“adjudication”, alongside standards and rules) is borrowed from our Western legal system, which manages to assign penalties for vague infractions like “reckless driving” and “disturbing the peace” despite its failure to provide any formal definition of these terms2. Meanwhile, “validation” is borrowed directly from the well-known concept of “verification and validation” from manufacturing and engineering-adjacent fields.

So, the main idea in this post is that systems for automating research can be developed to function analogously to some hybrid mix of law and systems engineering: Verification is what an AI does when formal rules are involved, adjudication is what an AI does when success is defined through (informal) standards, and validation3 is what an AI does to align standards with the higher-level aims of research.

Verify, adjudicate, validate
Figure 1: An AI output can undergo either verification or adjudication, with the latter being conducted according to pre-defined standards. These standards themselves are subject to validation.

An automated research system developed according to this framework will use formal verification whenever possible, but will otherwise work within your taste-based standards for making research progress, and develop any missing criteria (and validate such criteria) along the way. The remainder of this post will dive a bit more into the details of this setup, and then highlight some shortcomings of this approach.

Rules versus Standards

The distinction between verification and (what I am calling) adjudication runs parallel to the distinction between rules and standards. The former – rules – are specific and concrete requirements or imperatives, like “you must drive below 100 km/hr” or “worker radiation exposure must not exceed 10 mrem per consecutive 365 days”. In contrast, standards are all of the vague and imprecise provisions for policing behavior that is hard to define but that you “know it when you see it”. An example is a law prohibiting “disorderly conduct”, which (alongside clearer cases like “fighting” or obstructing traffic) might occur when someone “makes unreasonable noise”, engages in “tumultuous” behavior, or “disturbs any lawful assembly of persons”4.

It is not only remarkable that our justice system functions at all with laws written like this, it is inspirational for how we might extend our leverage as scientists using. Our surprisingly effectiveness legal system arises from many actors coordinating and making good-faith attempts to adhere to rules and standards that they cannot quite define rigorously.

Macroscopically, science already functions this way: Peer review involves submitting research for judgement by expert referees, who evaluate the work according to their own (unstated) standards for good research, and then an editor evaluates these evaluations by checking for invalid criteria for rejection (e.g. conflicts of interest or the presence of small typos) before making a final decision whether to accept the article into the journal. This system is famously effective at sorting out the best and correct ideas , and is almost universally liked by academics for its ability to reach decisions that leave all parties involved satisfied.

That was sarcasm. More seriously, developing standards and using adjudication in your own AI-driven research means constructing a system of review that will produce research that satisfies your individual or institutional goals. Much of this is an exercise in explicit communication: Are you able to state your goals? Would you know if these goals had been met? This suggests why improving AI capabilities cannot overcome the need for validation: The research tastes of an AI cannot simultaneously align with the preferences of a diverse crowd of researches who themselves disagree about the “right” questions to be asking or what constitutes an answer to said questions.

Adjudication all the way down

Moving beyond verification certainly has challenges. The first problem is that applying standards can be a many-layered process. You might think you established unambiguous standards, up until the point that your criteria were fatally misinterpreted. This is made worse when a research project originates from an underspecified question.

As an example: Suppose I have developed a new model (“model A”) for predicting the results of an experiment. I now pose a research/benchmarking question, “is model A a good fit for my experimental observations?”. Realizing I have provided far too little context, I follow up with some standards against which to judge AI outputs: “A model is ‘good’ if it is robust to sampling error, if it is data-efficient, and if it…”

Perhaps you can see where this is going. Eventually, in the process of spelling things out, I will feel that I have provided enough context and validated each criterion, definition, and requirement as being relevant and correct for my research. At this point, an AI will take over responsibility for applying these standards (reflecting my preferences), and in the act of applying these standards it will invoke its own preferences and capabilities. Continuing with the example, the AI might need to introduce additional criteria, like “a model is defined as robust if conditions {X, Y, Z} hold”. And then an additional layer of validation: (“is X the correct standard, or should it be W”?), and then introduce another layer of adjudication (“is X actually being met?”). Ideally, this process terminates with quantitative criteria that are formally verifiable (some statistical test for robustness, or some proof of average-case robustness), allowing the bots to finally do what they do best.

The less specific and comprehensive our initial criteria are, the more we will be relying on AI to supply intermediate standards for checking whether those criteria hold, and then schemes for validating those intermediate standards. In these cases, the risk is that the AI’s judgements are not sufficiently aligned with our own. After all, when there are infinitely many formally verifiable mathematical statements, verification is not enough.

AI as normal science

A second problem with this framework is that setting up systems for conducting research according to standards and rule-following will only ever achieve just that: research that follows a set of established rules. We can see parallels with “normal science”, a term coined by Thomas Kuhn (the philosopher who developed the idea of a scientific “paradigm”, later popularized in the phrase “paradigm shift”). Normal science is “…the attempt to force nature into conceptual boxes supplied by professional education”, boxes which are “preformed and relatively inflexible”5. While normal science and scientific paradigms originally described the broad framework for how scientists think about the physical world (rather than the preferences of some small group of researchers), the end result is similar: An adjudication-based system for automated research, being rooted in a concrete world view, is unlikely to shake up how we formalize and conceptualize the field we are conducting research in6.

Or more briefly, you should not be surprised if you are…not surprised by the results of your automated AI researcher.

  1. I am using law-inspired language to describe a what is going on in a variety of iterative, multi-agent harnesses/workflows, like generator-evaluator loops, ranking systems, and ensembling systems. My main motivation for this post was to come up with language to describe iterative systems when formal verification isn’t involved. ↩

  2. Obviously many people are unhappy about how some or another law is enforced, but I will take as a basic premise that the reader (i) believes that laws should exist but (ii) acknowledges that correct enforcement of these laws is not formally verifiable. ↩

  3. To reemphasize: validation does not mean to determine whether a statement is valid according to formal logic, which would actually be called “verification” in this post. ↩

  4. ORS 166.025 ↩

  5. Thomas S. Kuhn, The Structure of Scientific Revolutions, 2nd ed. (Chicago: University of Chicago Press, 1970) ↩

  6. The language of ‘rules’ and ‘standards’ was not foreign to Kuhn. He characterized research within a paradigm as follows: “men whose research is based on shared paradigms are committed to the same rules and standards for scientific practice” (italics mine). However, he was explicitly opposed to the idea that a paradigm was defined by rules and standards, and if the latter existed at all they would be understood to follow from an existing paradigm, e.g. “Paradigms may be prior to, more binding, and more complete than any set of rules for research that could be unequivocally abstracted from them.” ↩