Paper: arXiv:2609.19134Project pageGitHubModels

64executable environments
27scientific codebases, pinned and unmodified
2,812manufactured tasks
1,076scientific checks
85tasks in the ScienceIDE-Hard benchmark
Diagram of ScienceIDE: upstream scientific codes such as Athena++, Gkeyll, DScribe, NEST, MITgcm, EDKit and Stim feed environments; an editor view shows a task's instruction, the injected defect and its reference repair, a verifier table of physics cases, and SFT and RL panels
ScienceIDE in one view. Pinned upstream scientific codes (Athena++, MITgcm, Gkeyll, DScribe, NEST, EDKit, Stim, …) become containerized environments. A task injects a defect or removes a routine, and a verifier re-runs the code's own physics cases to grade the agent's repair. The same episodes feed supervised fine-tuning and reinforcement learning. From the ScienceIDE repository.

The idea

Scientific software holds decades of expertise, and it carries its own ground truth: the physics test cases a code must reproduce. ScienceIDE turns that into a training ground for AI agents. Each task is a containerized episode on a pinned, unmodified upstream code: astrophysical MHD, ocean and atmosphere models, plasma kinetics, quantum circuits, materials descriptors. The agent edits source. A verifier then recompiles the code, re-runs its physics cases and compares the numbers against reference output. The reward comes from the simulation being numerically right again, not from matching a diff, which makes it hard to game.

How a task is made

Five-step pipeline: real science code, propose a task, execute to validate, deterministic gates, release; rejected tasks go to a graveyard with their evidence
From code to released task. A defect is injected (repair) or a routine excised (implementation), and the reference fix is its exact inverse. Execution in the grading container must give the official fix a score of 1.0, while the unfixed build leaves room for a reward signal. Deterministic gates check that nothing leaks the answer. Difficulty is recorded as measured facts, not a hand label, and every rejected task is kept with its evidence. From the ScienceIDE repository.
  • Repair: a semantic defect is injected into the pinned source; the agent must find and fix it so the physics cases pass again.
  • Implementation: the body of a routine is removed; the agent reimplements it so the solver reproduces the reference results.
  • Grading: the reward is the mean physics-case score, rescaled so that an untouched repository earns exactly zero.

Three uses of one foundation

The same environments serve three purposes: evaluating agents (ScienceIDE-Hard), supervised fine-tuning on verified trajectories, and reinforcement learning from verifier rewards. The PhAI-IDE models (4B, 9B, 72B) are released on Hugging Face. One environment is LAPS, the UCLA pseudo-spectral MHD code behind my own simulations of Alfvénons.

My role

I am co-first author with Hejia Geng, and I co-led the generation of the scientific environments.

  • Geng, H.*, Huang, Z.*, Li, H., et al. (45 authors) 2026, “ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments.” arXiv:2609.19134 (* equal contribution)