Added caseHOLD task

[zolastro]

Hello,

I have been working on implementing caseHOLD within the lm-harness-evaluation framework. While I managed to create the necessary files (casehold.yaml and utils.py) to preprocess and evaluate the dataset, I encountered significant challenges during evaluation. Specifically, all models I tested consistently performed at or below random chance, which indicates an issue I have been unable to resolve.

Results with

lm_eval --model hf v--model_args pretrained=meta-llama/Llama-3.2-1B-Instruct --task casehold --device cuda:1 --batch_size 1 --apply_chat_template

| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------|------:|------|-----:|--------|---|-----:|---|------|
|casehold|      1|none  |     0|acc     |↑  |0.1472|±  |0.0049|
|        |       |none  |     0|acc_norm|↑  |0.2633|±  |0.0060|
|        |       |none  |     0|f1      |↑  |0.1472|±  |   N/A|

I’ve made this pull request to share my implementation and kindly request assistance in debugging or refining it. I’ve also opened an issue linked to this pull request to facilitate further discussion.

For context, here are the resources relevant to the task:

The caseHOLD dataset
Paper on caseHOLD
Any guidance or feedback would be greatly appreciated. Thank you in advance for your support!

Best regards,
David

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Antonio David Ponce Martínez seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[baberabb]

Hi! Have you tried prompting it in the MMLU style, where the choices are all in context and the loglikehood of the answer letter (A,B, etc.) is compared to determine the most probable response? might provide a better signal, as the choice holdings seem to be too subtle for a general model to distinguish

lm-evaluation-harness/lm_eval/tasks/mmlu/default/_default_template_yaml

Lines 7 to 8 in 8d2f64c

	doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
	doc_to_choice: ["A", "B", "C", "D"]