The most interesting change in OpenAI’s science work is not that models know more science. It is that the research program increasingly asks whether an AI system can do work a scientist can inspect: explore messy data, revise an analysis, maintain scientific software, produce a formal certificate, or expose a broken evaluation.
That is a higher bar than fact recall and a safer one than declaring “AI scientist” after one impressive answer. The current evidence spans four levels: benchmark judgment, agent-assisted field work, candidate discovery, and formal verification.
The newest claim: ten mathematical advances with formal certificates
On August 1, OpenAI published ten advances in mathematics and theoretical computer science. OpenAI says an internal version of a forthcoming model called Astra generated arguments for ten problems across geometry, coding theory, group theory, complexity, quantum games, lattice cryptography, and combinatorics.
People then prepared the manuscripts with the model, and the model formalized each argument as a Lean certificate. OpenAI explicitly attributes the mathematical arguments to the system while taking responsibility for correctness.
Three parts of that report deserve separate treatment:
- Generation: a model proposed the mathematical arguments.
- Editorial preparation: people worked with the model to turn those arguments into manuscripts.
- Formalization: Lean certificates provide machine-checkable verification that the encoded proofs follow from their assumptions.
Formal verification is strong evidence of logical validity for the formalized statement. It does not, by itself, establish that the statement captures every intended informal nuance, that the result is novel, or that the mathematical community will judge it important. Those questions require expert review and independent reproduction. The release is significant precisely because it supplies artifacts that make that scrutiny possible.
Scientific computing: implementation got cheaper, validation became the bottleneck
OpenAI’s July 28 field report, Scientific computing in the age of agentic AI, covers eight agent-assisted software projects, primarily in life sciences. Five used Codex alone and three used Codex with Claude Code. The work ranged from maintenance and packaging to language migrations and GPU-oriented redesigns.
Contributors reported faster implementation, but they also reported confident errors, difficult numerical edge cases, and a long last mile. The strongest projects used external references or measurable acceptance targets: exact output parity, known statistical behavior, simulated data, or established answers.
The report is exploratory and retrospective. It is not a randomized productivity study. Its recurring pattern is still useful:
| Before agents | With coding agents | New bottleneck |
|---|---|---|
| Researcher implements and verifies | Agent implements; researcher specifies and verifies | Reliable validation |
| Maintenance work is deferred | More rewrites and migrations become affordable | Stewardship and ownership |
| One implementation path dominates | Many alternatives can be generated | Fragmentation and long-term support |
| Engineering skill limits iteration | Domain judgment limits release | Scientific acceptance criteria |
Lower implementation cost does not remove responsibility. It shifts expert effort toward specifying invariants, selecting evidence, validating results, and maintaining what ships.
GeneBench-Pro: testing research taste under controlled truth
GeneBench-Pro targets computational biology problems where the difficult part is choosing an analysis under ambiguity. Each task provides a messy dataset, experimental context, and a target estimand tied to a decision. The agent must explore, choose a method, revise when diagnostics contradict assumptions, and return a result.
The benchmark contains 129 questions built from synthetic data-generating processes. Synthetic construction is a deliberate evaluation design: the creators know the causal structure and target answer, so they can grade deterministically while still injecting realistic quality problems and confounding.
OpenAI reports GPT-5.6 Sol passing 28.7% at its highest reasoning level and 31.5% in Pro mode. That is not autonomous biology. It is evidence that a measurable frontier now exists for multistage statistical judgment—and that most tasks remain unsolved.
LifeSciBench: expert usefulness, not one clean answer
LifeSciBench takes a complementary approach. It contains 750 expert-authored tasks, 1,062 artifacts, 19,020 rubric criteria, 173 scientist contributors, and 453 independent expert reviewers across real life-science workflows.
Tasks cover evidence handling, analysis, design and optimization, reasoning, validation and operations, translation, and scientific communication. More than half require interpreting at least one artifact; most require multiple reasoning or decision steps.
The detailed rubrics matter. Scientific usefulness often depends on caveats, methods, and downstream consequences, not only the final sentence. A model can reach the right top-level conclusion while missing an assay limitation that makes the recommendation unsafe.
OpenAI positions the benchmark as task-level evidence, not proof of downstream research impact. The article explicitly calls for live deployment studies over longer horizons and experimental feedback loops. That restraint is the correct boundary.
Eval research is part of the science program
Two recent OpenAI studies show why better tasks are not enough; the harness and dataset need their own evaluation.
In Separating signal from noise in coding evaluations, an agent-assisted audit plus independent review by experienced engineers estimated that roughly 30% of the public SWE-Bench Pro tasks were broken. Failure modes included overly strict tests, underspecified prompts, low test coverage, and misleading instructions. OpenAI retracted its earlier recommendation to adopt the benchmark.
In the ARC-AGI-3 harness analysis, retaining reasoning and using compaction instead of rolling truncation roughly tripled GPT-5.6 Sol’s score on the public set while reducing output tokens. The same model under a different context policy behaved like a materially different system.
Together these studies support a hard rule: an evaluation result belongs to a tuple, not a model.
result = model
+ prompt
+ tools
+ context policy
+ runtime settings
+ task data
+ grader
+ retry and stopping policyIf any element changes, the result is a new experimental condition.
An evidence ladder for AI-assisted science
Use this ladder when assessing a research claim:
| Level | Evidence | What it supports |
|---|---|---|
| 1 | Curated benchmark score | Capability on the benchmark distribution |
| 2 | Artifact-rich expert evaluation | Usefulness on realistic task fragments |
| 3 | Retrospective field report | Feasibility and observed workflow patterns |
| 4 | Prospective controlled study | Causal impact on time, quality, or discovery |
| 5 | Reproducible external result | Independent scientific validity |
| 6 | Formal certificate plus expert review | Logical verification and domain interpretation |
| 7 | Durable adoption with stewardship | Ongoing scientific and operational value |
No single OpenAI release spans the whole ladder. The mathematics work reaches formalization but awaits community assessment. GeneBench-Pro and LifeSciBench deepen benchmark realism. The scientific-computing report shows field use without causal attribution. Taken together, they show a program moving toward inspectable and verifiable research contribution.
A checklist for deploying agents in science
- Define the scientific decision, not just the requested artifact.
- Preserve raw data, code, environment, prompts, tool versions, and traces.
- Use simulated data or known references when a deterministic target is possible.
- Separate implementation review from scientific-validity review.
- Require uncertainty, assumptions, and failed analyses in the record.
- Validate numerical parity and edge cases before optimizing speed.
- Name a long-term steward before releasing generated scientific software.
- Distinguish a model-produced hypothesis from a verified result.
- Publish enough artifacts for independent reproduction.
The most credible vision of AI for science is not a model replacing scientific judgment. It is a system making more candidate work possible while making verification, attribution, and stewardship impossible to ignore.