ChemPro

A progressive chemistry benchmark for large language models.

ChemPro is an evaluation benchmark. It contains 4,100 chemistry questions in four tiers of increasing difficulty.

Most benchmarks take their difficulty labels from annotator judgement. ChemPro does not. Each tier comes from one established educational source:

Decades of curriculum design set the difficulty of each tier. No one applied the labels afterwards.

The benchmark keeps conceptual understanding separate from computational reasoning. Multiple-choice questions test the first. Numerical questions test the second. You can score the two independently.

Why the tiers matter

Many models score well on introductory chemistry. Their accuracy then drops sharply as the questions become harder to read, even when the concepts stay inside the same syllabus.

JEE Mains follows the official NCERT syllabus. The challenging and difficult tiers therefore cover the same concepts. The accuracy gap between these two tiers shows the effect of question complexity alone, not of subject coverage.

Contents

Split tier Source MCQ Numerical Total
easy CP_E Web quizzes and questionnaires 590 204 794
medium CP_M NCERT, grades 9–10 229 107 336
challenging CP_C NCERT, grades 11–12 455 208 663
difficult CP_D JEE Mains, 2020–2024 1,493 814 2,307
Total     2,767 1,333 4,100

Each question also has one or more subfield labels:

Subfield Items
Inorganic-Chemistry 1,651
Physical-Chemistry 1,536
Organic-Chemistry 915
Bio-Chemistry 227

Usage

from datasets import load_dataset

# One tier of multiple-choice questions
mcq = load_dataset("sochastic/ChemPro", "mcq", split="difficult")
print(mcq[0]["question"])
print(mcq[0]["choices"])
print(mcq[0]["answer"], mcq[0]["answer_text"])

# All tiers at once
every_tier = load_dataset("sochastic/ChemPro", "mcq")

# Numerical problems
num = load_dataset("sochastic/ChemPro", "numerical", split="easy")
print(num[0]["question"], num[0]["answer_value"], num[0]["answer_unit"])

To filter by subfield:

organic = mcq.filter(lambda r: "Organic-Chemistry" in r["attributes"])

Schema

Both configs share these fields:

Field Type Description
id string A stable identifier, for example chempro-difficult-mcq-01504. You can cite it.
source_id int32 The index of the question in its source tier and question type
tier string The tier symbol used in the paper: CP_E, CP_M, CP_C or CP_D. The split name gives the readable form.
source string web, ncert_grade_9_10, ncert_grade_11_12 or jee_mains_2020_2024
question_type string mcq or numerical
question string The question stem. For MCQ items, the options are in choices.
attributes list[string] One or more of the four subfields above

The mcq config adds these fields:

Field Type Description
choices list[string] Exactly four options, in the order A to D
answer string The letter of the correct option, A to D
answer_index int32 The zero-based index of the correct option in choices
answer_text string The text of the correct option. It always equals choices[answer_index].

The numerical config adds these fields:

Field Type Description
answer string The answer as written. It includes the unit when the source gave one.
answer_value float64 The numeric value taken from answer. Use it for tolerance-based scoring.
answer_unit string The unit taken from answer. It is null when the answer has no unit.

Examples

An MCQ item:

{
  "id": "chempro-difficult-mcq-00001",
  "tier": "CP_D",
  "source": "jee_mains_2020_2024",
  "question_type": "mcq",
  "question": "The five successive ionization enthalpies of an element are ...",
  "choices": ["2", "4", "3", "5"],
  "answer": "C",
  "answer_index": 2,
  "answer_text": "3",
  "attributes": ["Inorganic-Chemistry"]
}

A numerical item:

{
  "id": "chempro-easy-numerical-00001",
  "tier": "CP_E",
  "question_type": "numerical",
  "question": "Calculate the molar mass of $$ CH_3COOH $$.",
  "answer": "60.05 g/mol",
  "answer_value": 60.05,
  "answer_unit": "g/mol",
  "attributes": ["Physical-Chemistry"]
}

Chemical notation stays in LaTeX between $$ delimiters. Remove the delimiters or render them, as your evaluation requires.

Evaluation notes

Construction

We took the questions from the sources above. We put them into one text format. Three passes then verified them:

  1. A source check confirmed the origin of each question.
  2. An expert review confirmed that each question is correct and has one meaning.
  3. Automated checks looked for format errors and duplicates.

GPT-4o applied the subfield attributes. Human annotators did not. Use these labels to slice the data. Do not use them as a gold-standard taxonomy.

Each tier holds its full number of questions. Every question matches the difficulty level and the subject mix of its tier.

Limitations

Licensing and provenance

ChemPro uses the CC BY-NC 4.0 licence. You can share and adapt the dataset for non-commercial purposes. You must give attribution.

The medium and challenging tiers come from NCERT curricular material. The difficult tier comes from JEE Mains examination papers. The National Testing Agency administers those examinations. We adapted the questions. We did not copy them word for word. The non-commercial term follows from these sources. If you plan to use the dataset for more than non-commercial research, check the status of the source material first.

Citation

If you use ChemPro, please cite the journal article:

@article{baranwal2026chempro,
  title   = {ChemPro: A progressive chemistry benchmark for Large Language Models},
  author  = {Baranwal, Aaditya and Vyas, Shruti},
  journal = {Artificial Intelligence Chemistry},
  volume  = {4},
  number  = {1},
  pages   = {100118},
  year    = {2026},
  issn    = {2949-7477},
  doi     = {10.1016/j.aichem.2026.100118},
  url     = {https://www.sciencedirect.com/science/article/pii/S2949747726000126}
}

The preprint:

@article{baranwal2026chempro_arxiv,
  title         = {ChemPro: A Progressive Chemistry Benchmark for Large Language Models},
  author        = {Baranwal, Aaditya and Vyas, Shruti},
  journal       = {arXiv preprint arXiv:2602.03108},
  year          = {2026},
  doi           = {10.48550/arXiv.2602.03108},
  url           = {https://arxiv.org/abs/2602.03108}
}

Contact

Aaditya Baranwal, University of Central Florida. Email: aaditya.baranwal@ucf.edu