Skip to content

Commit

Permalink
add Basque translation of PIQA (piqa_eu) to BasqueBench (#2531)
Browse files Browse the repository at this point in the history
  • Loading branch information
naiarapm authored Dec 3, 2024
1 parent 1170ef9 commit f49b037
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 0 deletions.
2 changes: 2 additions & 0 deletions lm_eval/tasks/basque_bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The new evaluation datasets included in BasqueBench are:
| Task | Category | Homepage |
|:-------------:|:-----:|:-----:|
| MGSM_eu | Math | https://huggingface.co/datasets/HiTZ/MGSM-eu |
| PIQA_eu | Question Answering | https://huggingface.co/datasets/HiTZ/PIQA-eu |
| WNLI_eu | Natural Language Inference | https://huggingface.co/datasets/HiTZ/wnli-eu |
| XCOPA_eu | Commonsense Reasoning | https://huggingface.co/datasets/HiTZ/XCOPA-eu |

Expand Down Expand Up @@ -63,6 +64,7 @@ The following tasks evaluate tasks on BasqueBench dataset using various scoring
- `flores_pt-eu`
- `mgsm_direct_eu`
- `mgsm_native_cot_eu`
- `piqa_eu`
- `qnlieu`
- `wnli_eu`
- `xcopa_eu`
Expand Down
1 change: 1 addition & 0 deletions lm_eval/tasks/basque_bench/basque_bench.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,6 @@ task:
- xcopa_eu
- mgsm_direct_eu
- mgsm_native_cot_eu
- piqa_eu
metadata:
version: 1.0
21 changes: 21 additions & 0 deletions lm_eval/tasks/basque_bench/piqa_eu.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
task: piqa_eu
dataset_path: HiTZ/PIQA-eu
dataset_name: null
output_type: multiple_choice
training_split: null
validation_split: validation
test_split: null
doc_to_text: "Galdera: {{goal}}\nErantzuna:"
doc_to_target: label
doc_to_choice: "{{[sol1, sol2]}}"
should_decontaminate: true
doc_to_decontamination_query: goal
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
metadata:
version: 1.0

0 comments on commit f49b037

Please sign in to comment.