Table of Contents
Shortcuts

Dataset Quick Evaluation Tutorial

OpenCompass provides two ways to quickly evaluate your own data, based on the CustomDataset and ChatMLDataset data format protocols, respectively. Compared with the complete dataset integration process in new_dataset.md, these two evaluation paths are more convenient and allow evaluation without implementing a new Dataset class. But if you have specific needs for custom reading/inference/evaluation, it is recommended to still follow the complete integration process to add a new dataset.

Data Format Protocol and Fast Evaluation Based on CustomDataset

(This module is no longer being updated, but it can still be used for quick evaluation from the command line.)

This module supports two types of tasks: multiple choice (mcq) and question answering (qa). For mcq, both ppl and gen inference are supported; for qa, gen inference is supported.

Dataset Format

We support datasets in both .jsonl and .csv formats.

Multiple Choice (mcq)

For mcq datasets, the default fields are as follows:

  • question: The stem of the multiple-choice question.

  • A, B, C, …: Single uppercase letters from A to Z, supporting at most 26 options. The options must be consecutive and start from A.

  • answer: The correct answer to the multiple-choice question, which must be one of the options used above, such as A, B, C, etc.

Non-default fields will be read in but are not used by default. To use them, specify in the .meta.json file.

An example of the .jsonl format:

{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer": "B"}
{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880", "answer": "A"}
{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C": "5383", "answer": "B"}
{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C": "4178", "answer": "C"}

An example of the .csv format:

question,A,B,C,answer
127+545+588+620+556+199=,2632,2635,2645,B
735+603+102+335+605=,2376,2380,2410,B
506+346+920+451+910+142+659+850=,4766,4774,4784,C
504+811+870+445=,2615,2630,2750,B

Question & Answer (qa)

For qa datasets, the default fields are as follows:

  • question: The question to answer.

  • answer: The correct answer. It can be omitted when the dataset has no reference answer.

Non-default fields will be read in but are not used by default. To use them, specify in the .meta.json file.

An example of the .jsonl format:

{"question": "752+361+181+933+235+986=", "answer": "3448"}
{"question": "712+165+223+711=", "answer": "1811"}
{"question": "921+975+888+539=", "answer": "3323"}
{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"}

An example of the .csv format:

question,answer
123+147+874+850+915+163+291+604=,3967
149+646+241+898+822+386=,3142
332+424+582+962+735+798+653+214=,4700
649+215+412+495+220+738+989+452=,4170

Command Line List

Custom datasets can be directly called for evaluation through the command line.

opencompass \
    --models hf_llama2_7b \
    --custom-dataset-path xxx/test_mcq.csv \
    --custom-dataset-data-type mcq \
    --custom-dataset-infer-method ppl

opencompass \
    --models hf_llama2_7b \
    --custom-dataset-path xxx/test_qa.jsonl \
    --custom-dataset-data-type qa \
    --custom-dataset-infer-method gen

In most cases, --custom-dataset-data-type and --custom-dataset-infer-method can be omitted. OpenCompass sets them based on the following logic:

  • If at least two consecutive options starting from A (such as A, B, and C) can be parsed from the dataset file, it is considered an mcq dataset; otherwise, it is considered a qa dataset.

  • The default infer_method is gen.

Configuration File

In the original configuration file, simply add a new item to the datasets variable. Custom datasets can be mixed with regular datasets.

datasets = [
    {"path": "xxx/test_mcq.csv", "data_type": "mcq", "infer_method": "ppl"},
    {"path": "xxx/test_qa.jsonl", "data_type": "qa", "infer_method": "gen"},
]

Supplemental Information for Dataset .meta.json

OpenCompass will try to parse the input dataset file by default, so in most cases, the .meta.json file is not necessary. However, if the dataset field names are not the default ones, or custom prompt words are required, it should be specified in the .meta.json file.

The file is placed in the same directory as the dataset, with the filename followed by .meta.json. An example file structure is as follows:

.
├── test_mcq.csv
├── test_mcq.csv.meta.json
├── test_qa.jsonl
└── test_qa.jsonl.meta.json

Possible fields in this file include:

  • abbr (str): Abbreviation of the dataset, serving as its ID.

  • data_type (str): Type of dataset, options are mcq and qa.

  • infer_method (str): Inference method, options are ppl and gen.

  • human_prompt (str): User prompt template for generating prompts. Variables in the template are enclosed in {}, like {question}, {opt1}, etc. If template exists, this field will be ignored.

  • bot_prompt (str): Bot prompt template for generating prompts. Variables in the template are enclosed in {}, like {answer}, etc. If template exists, this field will be ignored.

  • template (str or dict): Question template for generating prompts. Variables are enclosed in {}, such as {question} and {opt1}. For the relevant syntax, see Legacy Prompt Template.

  • input_columns (list): List of input fields for reading data.

  • output_column (str): Output field for reading data.

  • options (list): List of options for reading data, valid only when data_type is mcq.

For example:

{
    "human_prompt": "Question: 127 + 545 + 588 + 620 + 556 + 199 =\nA. 2632\nB. 2635\nC. 2645\nAnswer: Let's think step by step, 127 + 545 + 588 + 620 + 556 + 199 = 672 + 588 + 620 + 556 + 199 = 1260 + 620 + 556 + 199 = 1880 + 556 + 199 = 2436 + 199 = 2635. So the answer is B.\nQuestion: {question}\nA. {A}\nB. {B}\nC. {C}\nAnswer: ",
    "bot_prompt": "{answer}"
}

or

{
    "template": "Question: {my_question}\nX. {X}\nY. {Y}\nZ. {Z}\nW. {W}\nAnswer:",
    "input_columns": ["my_question", "X", "Y", "Z", "W"],
    "output_column": "my_answer"
}

Data Format Protocol and Fast Evaluation Based on ChatMLDataset

OpenCompass has recently launched a dataset evaluation mode based on the ChatML dialogue template, which allows users to provide a .jsonl dataset file that conforms to the ChatML dialogue template, and simply set the dataset information config like model configs to start evaluating directly.

Format Requirements for Data Files

This evaluation method only supports data files in .jsonl format, and each sample must comply with the following format:

The format of a text-only dataset with a simple structure:

{"question":[{"role":"user","content":"What is 165 + 833 + 650 + 615?"}],"answer":["2263"]}
{"question":[{"role":"user","content":"What is the capital of China?"}],"answer":["Beijing"]}

Each line must be a complete JSON object. The current quick evaluation path only uses the first user message and the first item in answer. It does not support multi-turn conversations or use system messages.

When ChatMLDataset reads .jsonl files, it will use pydantic to perform simple format validation on the files. You can use tools/chatml_format_test.py to check your provided data file.

python tools/chatml_format_test.py --path /path/to/dataset.jsonl

After format checking, define datasets = [] and add a configuration list named chatml_datasets to the run configuration so that OpenCompass can convert the data file into a dataset at runtime. An example is as follows:

datasets = []

chatml_datasets = [
    dict(
        abbr='YOUR_DATASET_NAME',
        path='YOUR_DATASET_PATH',
        evaluator=dict(
            type='cascade_evaluator',
            rule_evaluator=dict(
                type='math_evaluator',
            ),
            llm_evaluator=dict(
                type='llm_evaluator',
                prompt="YOUR_JUDGE_PROMPT",
                judge_cfg=dict(), # YOUR Judge Model Config
            )
        ),
        n=1, # Repeat Number
    ),
]

After saving the configuration file, start the evaluation with:

opencompass /path/to/config.py

The ChatML evaluation module currently provides four preset evaluators: mcq_rule_evaluator for MCQ evaluation, math_evaluator for LaTeX mathematical formula evaluation, llm_evaluator for evaluating answers that are open-ended or difficult to extract, and cascade_evaluator, which combines rule-based and LLM evaluators in a cascade.

In addition, if you have a long-term need to use datasets based on ChatML templates, you can contribute your dataset config to opencompass/configs/chatml_datasets. An eval example of calling these dataset configs is provided in examples/eval_chatml_datasets.py.

@沪ICP备2021009351号-23 OpenCompass Open Platform Service Agreement