#
Offline evaluation using local data files

If you are prototyping a task based on files that are not yet hosted on the Hub, you can take advantage of the `hf_data_files` argument to point lighteval at local JSON/CSV resources. This makes it easy to evaluate datasets that live in your repo or that are generated on the fly.

Internally, `hf_data_files` is passed directly to the `data_files` parameter of `datasets.load_dataset` (docs).

See adding a custom task for more information on how to create a custom task.

    from pathlib import Path

    from lighteval.metrics import Metrics
    from lighteval.tasks.lighteval_task import LightevalTaskConfig
    from lighteval.tasks.requests import Doc

    def local_prompt(line: dict, task_name: str) -> Doc:
        return Doc(
            task_name=task_name,
            query=line["question"],
            choices=line["choices"],
            gold_index=line["answer"]
        )

    local_data = Path(__file__).parent / "samples" / "faq.jsonl"

    local_task = LightevalTaskConfig(
        name="faq_eval",
        prompt_function=local_prompt,
        hf_repo="json",  # Built-in streaming loader for json/jsonl files
        hf_subset="default",
        hf_data_files=str(local_data),  # Can also be a dict mapping split names to paths
        evaluation_splits=["train"],
        metrics=[Metrics.ACCURACY],
    )

Once the config is registered in `TASKS_TABLE`, running the task with `--custom-tasks path/to/your_file.py` will automatically load the local data files. You can also pass a dictionary to `hf_data_files` (e.g. `{"train": "train.jsonl", "validation": "val.jsonl"}`) to expose multiple splits.

Update on GitHub