ChemPro

ChemPro: A Progressive Chemistry Benchmark for Large Language Models

Paper arXiv Dataset

Published in Artificial Intelligence Chemistry, volume 4, issue 1, article 100118 (2026).

ChemPro contains 4,100 chemistry questions in four tiers of increasing difficulty. Each tier comes from one established educational source, so the difficulty comes from curriculum design and not from labels applied afterwards. Multiple-choice questions test conceptual understanding. Numerical questions test computational reasoning. You can score the two separately.

This repository holds the evaluation code and the pipeline that built the released dataset. The data itself is on the Hugging Face Hub at sochastic/ChemPro.

Split Source MCQ Numerical Total
easy Web quizzes and questionnaires 590 204 794
medium NCERT, grades 9–10 229 107 336
challenging NCERT, grades 11–12 455 208 663
difficult JEE Mains, 2020–2024 1,493 814 2,307
Total   2,767 1,333 4,100

Each question also carries one or more subfield labels: Inorganic, Physical, Organic or Bio-Chemistry. The dataset card documents the schema, the construction and the limitations.

Load the data

from datasets import load_dataset

mcq = load_dataset("sochastic/ChemPro", "mcq", split="difficult")
num = load_dataset("sochastic/ChemPro", "numerical", split="easy")
print(mcq[0]["question"], mcq[0]["choices"], mcq[0]["answer"])
print(num[0]["question"], num[0]["answer_value"], num[0]["answer_unit"])

Evaluate a model

The evaluation runs a Hugging Face model with vLLM, so it needs a CUDA GPU.

pip install -r requirements.txt

python evaluate.py -m Qwen/Qwen3-8B -t mcq
python evaluate.py -m Qwen/Qwen3-8B -t numerical --splits easy,medium -p 2
Option Meaning
-m, --model Hugging Face model id or local path
-t, --type mcq or numerical
--splits Comma-separated tiers. The default is all four.
-p, --parallel Tensor-parallel size (number of GPUs)
--max-tokens Generation limit per question. The default is 1000. Reasoning models may need more.
--data-dir Read the parquet files from a local copy instead of the Hub
--dry-run Load the data and print the first prompt of each split, without loading a model

For each split, the script writes these files to results/{model}/{type}/:

runfiles/run_eval.sh is a SLURM template that evaluates several models in one job.

Scoring

The system prompt asks the model to write a solution and then give its answer under FINAL ANSWER:. The model is prompted with a ChatML template and greedy decoding.

Answer extraction:

Metrics:

Question type Metric Definition
MCQ overall_accuracy Fraction of predictions that match the correct letter
MCQ option_wise_accuracy Accuracy for each correct letter
MCQ answer_correlation Count table of correct letter against predicted letter
Numerical Exact match Prediction equals the answer value
Numerical Tolerance-based accuracy Prediction is within 10% of the answer: \|ŷ − y\| ≤ 0.1·\|y\|
Numerical Mean absolute error Average of \|ŷ − y\|
Numerical Mean relative error Average of \|ŷ − y\| / (\|y\| + \|ŷ\|)

Report the result for each tier. A single average hides the drop in accuracy that the benchmark is built to show.

The published release differs slightly from the item set scored in the paper. It repairs 155 items, removes 144 duplicates and adds 148 items to keep each tier at its original size. chempro-release/rectifications.json records every content change with its reason.

Run the unit tests with pytest. They need no GPU.

Release pipeline

chempro-release/ contains the dataset card, the dataset licence and the scripts that built the release. The build has no model calls and no randomness, so it produces byte-identical parquet files each time. VALIDATION_REPORT.md holds the combined checksum.

To check the published files against the validation rules, download them and run the validator. It exits non-zero if any check fails.

hf download sochastic/ChemPro --repo-type dataset --include "data/*" --local-dir chempro-release
python chempro-release/scripts/validate_release.py

A rebuild from source needs the pre-release question files in JSON/ at the repository root. Those files are not distributed. With them in place, the steps are:

python chempro-release/scripts/make_worklist.py         # list defective items -> worklist.json
python chempro-release/scripts/author_rectifications.py # write the repairs     -> rectifications.json
python chempro-release/scripts/extract_recovered.py     # recover lost items    -> recovered.json
python chempro-release/scripts/build_release.py         # -> chempro-release/data/*/*.parquet
python chempro-release/scripts/validate_release.py      # -> VALIDATION_REPORT.md

chempro_core.py holds the shared code for parsing, normalization, deduplication and defect detection. The same code that finds a defect also proves in validation that the defect is gone.

Repository layout

evaluate.py           evaluation entry point (vLLM)
chempro/              data loading, prompts, answer extraction, metrics
runfiles/             SLURM job template
tests/                unit tests for extraction and scoring
chempro-release/      dataset card, dataset licence, rectifications, build and validation scripts

Licence

The medium and challenging tiers are adapted from NCERT curricular material. The difficult tier is adapted from JEE Mains papers. The non-commercial term follows from these sources.

Citation

@article{baranwal2026chempro,
  title   = {ChemPro: A progressive chemistry benchmark for Large Language Models},
  author  = {Baranwal, Aaditya and Vyas, Shruti},
  journal = {Artificial Intelligence Chemistry},
  volume  = {4},
  number  = {1},
  pages   = {100118},
  year    = {2026},
  issn    = {2949-7477},
  doi     = {10.1016/j.aichem.2026.100118},
  url     = {https://www.sciencedirect.com/science/article/pii/S2949747726000126}
}

@article{baranwal2026chempro_arxiv,
  title   = {ChemPro: A Progressive Chemistry Benchmark for Large Language Models},
  author  = {Baranwal, Aaditya and Vyas, Shruti},
  journal = {arXiv preprint arXiv:2602.03108},
  year    = {2026},
  doi     = {10.48550/arXiv.2602.03108},
  url     = {https://arxiv.org/abs/2602.03108}
}

Contact: Aaditya Baranwal, University of Central Florida (aaditya.baranwal@ucf.edu)