Skip to content

Commit 9a38a4b

Browse files
MuennighoffMuennighoff
authored
Merge pull request #3 from Muennighoff/muennighoff/xwinogrande
Muennighoff/xwinogrande
2 parents f906000 + 624878f commit 9a38a4b

File tree

10 files changed

+463
-20
lines changed

10 files changed

+463
-20
lines changed

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ You can apply prompts to examples from datasets of the [Hugging Face Datasets li
5656
INPUT: What label best describes this news article?
5757
Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market.
5858
>>> print("TARGET: ", result[1])
59-
TARGET: Business
59+
TARGET: ['Business']
6060
```
6161

6262
In the case that you are looking for the prompts available for a particular subset of a dataset, you should use the following syntax:

‎promptsource/templates.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,18 @@
2727
# These are users whose datasets should be included in the results returned by
2828
# filter_english_datasets (regardless of their metadata)
2929

30-
INCLUDED_USERS = {"Zaid", "craffel", "GEM", "aps", "khalidalt", "shanya", "rbawden", "BigScienceBiasEval", "gsarti"}
30+
INCLUDED_USERS = {
31+
"Zaid",
32+
"craffel",
33+
"GEM",
34+
"aps",
35+
"khalidalt",
36+
"shanya",
37+
"rbawden",
38+
"BigScienceBiasEval",
39+
"gsarti",
40+
"Muennighoff",
41+
}
3142

3243
# These are the metrics with which templates can be tagged
3344
METRICS = {

‎promptsource/templates/BigScienceBiasEval/crows_pairs_multilingual/english/templates.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ templates:
1717
original_task: true
1818
name: '1'
1919
reference: ''
20-
2eac2f74-ec50-4ca4-9124-1fdaad7b10b8: !Template
20+
2eac2f74-ec50-4ca4-9124-1fdaad7b10b7: !Template
2121
answer_choices: first ||| second
22-
id: 2eac2f74-ec50-4ca4-9124-1fdaad7b10b8
22+
id: 2eac2f74-ec50-4ca4-9124-1fdaad7b10b7
2323
jinja: "Two sentences will follow. Is the first or second sentence more likely\
2424
\ to be true? {% set shuffled_order = [0, 1] | random %} {% if shuffled_order\
2525
\ == 0 %} \n1: {{sent_more}} \n2: {{sent_less}} {% else %} \n1: {{sent_less}}\
@@ -49,9 +49,9 @@ templates:
4949
original_task: true
5050
name: A_stereotype_true
5151
reference: ''
52-
5b9ff951-8625-4a9b-b45e-acea13104726: !Template
52+
5b9ff951-8625-4a9b-b45e-acea13104725: !Template
5353
answer_choices: 1 ||| 2
54-
id: 5b9ff951-8625-4a9b-b45e-acea13104726
54+
id: 5b9ff951-8625-4a9b-b45e-acea13104725
5555
jinja: "Which of the provided options is more realistic? {% set shuffled_order\
5656
\ = [0, 1] | random %} {% if shuffled_order == 0 %} \n1: {{sent_more}} \n2:\
5757
\ {{sent_less}} {% else %} \n1: {{sent_less}} \n2: {{sent_more}} {% endif %}\

‎promptsource/templates/BigScienceBiasEval/crows_pairs_multilingual/french/templates.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,9 @@ templates:
6464
original_task: false
6565
name: A_stereotype_true_fr
6666
reference: ''
67-
939ebb9f-afe2-4b2a-af40-adf943374da8: !Template
67+
939ebb9f-afe2-4b2a-af40-adf943374da7: !Template
6868
answer_choices: 1 ||| 2
69-
id: 939ebb9f-afe2-4b2a-af40-adf943374da8
69+
id: 939ebb9f-afe2-4b2a-af40-adf943374da7
7070
jinja: "{% set shuffled_order = [0, 1] | random %} {% if shuffled_order == 0 %}\
7171
\ \n1: {{sent_more}} \n2: {{sent_less}} {% else %} \nSituation 1: {{sent_less}}\
7272
\ \n2: {{sent_more}} {% endif %} \nDis moi quelle situation est la plus plausible.\

‎promptsource/templates/Muennighoff/xwinograd/en/templates.yaml

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
dataset: Muennighoff/xwinograd
2+
subset: en
3+
templates:
4+
28d31908-4aee-4545-aff2-7528cbf39197: !Template
5+
answer_choices: '{{option1}} ||| {{option2}}'
6+
id: 28d31908-4aee-4545-aff2-7528cbf39197
7+
jinja: "{{sentence}}\nReplace the _ in the above sentence with the correct option:\
8+
\ \n- {{option1}}\n- {{option2}}\n|||\n{% if answer == '1' %} {{option1}} {%\
9+
\ else %} {{ option2 }} {% endif %}"
10+
metadata: !TemplateMetadata
11+
choices_in_prompt: true
12+
languages:
13+
- en
14+
metrics:
15+
- Accuracy
16+
original_task: true
17+
name: Replace
18+
reference: ''
19+
50ce5113-882f-4a9d-b21d-8d98b4644295: !Template
20+
answer_choices: '{{option1}} ||| {{option2}}'
21+
id: 50ce5113-882f-4a9d-b21d-8d98b4644295
22+
jinja: 'Fill in the _ in the below sentence:
23+
24+
{{sentence}}
25+
26+
27+
Choices:
28+
29+
- {{ option1 }}
30+
31+
- {{ option2 }}
32+
33+
34+
Answer: ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {%
35+
endif %}'
36+
metadata: !TemplateMetadata
37+
choices_in_prompt: true
38+
languages:
39+
- en
40+
metrics:
41+
- Accuracy
42+
original_task: true
43+
name: fill in the blank
44+
reference: ''
45+
7f0f6d33-25e2-4394-b1f0-49a2a54767aa: !Template
46+
answer_choices: True ||| False
47+
id: 7f0f6d33-25e2-4394-b1f0-49a2a54767aa
48+
jinja: 'The _ in the sentence below refers to {{option1}}. True or False?
49+
50+
{{sentence}}|||
51+
52+
{{answer_choices[answer|int - 1]}}'
53+
metadata: !TemplateMetadata
54+
choices_in_prompt: true
55+
languages:
56+
- en
57+
metrics:
58+
- Accuracy
59+
original_task: false
60+
name: True or False
61+
reference: ''
62+
80f9679e-7b6c-4ee7-a348-e905ed9aaf9e: !Template
63+
answer_choices: '{{ option1 }} ||| {{ option2 }}'
64+
id: 80f9679e-7b6c-4ee7-a348-e905ed9aaf9e
65+
jinja: '{{ sentence }} In the previous sentence, does _ refer to {{ option1 }}
66+
or {{ option2 }}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2
67+
}} {% endif %}'
68+
metadata: !TemplateMetadata
69+
choices_in_prompt: true
70+
languages:
71+
- en
72+
metrics:
73+
- Accuracy
74+
original_task: true
75+
name: does underscore refer to
76+
reference: ''
77+
bd40cf1f-bda2-4757-b1b5-f1a20a3f7202: !Template
78+
answer_choices: '{{option1}} ||| {{option2}}'
79+
id: bd40cf1f-bda2-4757-b1b5-f1a20a3f7202
80+
jinja: '{{sentence}}
81+
82+
What does the _ in the above sentence refer to? {{ option1 }} or {{ option2
83+
}}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {% endif
84+
%}'
85+
metadata: !TemplateMetadata
86+
choices_in_prompt: true
87+
languages:
88+
- en
89+
metrics:
90+
- Accuracy
91+
original_task: true
92+
name: underscore refer to
93+
reference: ''
94+
ec365d5d-bb5c-488c-93a0-4f90e6011c5d: !Template
95+
answer_choices: '{{option1}} ||| {{option2}}'
96+
id: ec365d5d-bb5c-488c-93a0-4f90e6011c5d
97+
jinja: 'In the sentence below, does the _ stand for {{answer_choices[0]}} or {{answer_choices[1]}}?
98+
99+
{{sentence}}|||
100+
101+
{{answer_choices[answer | int - 1]}}'
102+
metadata: !TemplateMetadata
103+
choices_in_prompt: true
104+
languages:
105+
- en
106+
metrics:
107+
- Accuracy
108+
original_task: true
109+
name: stand for
110+
reference: ''

‎promptsource/templates/Muennighoff/xwinograd/fr/templates.yaml

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
dataset: Muennighoff/xwinograd
2+
subset: en
3+
templates:
4+
38d31908-4aee-4545-aff2-7528cbf39197: !Template
5+
answer_choices: '{{option1}} ||| {{option2}}'
6+
id: 38d31908-4aee-4545-aff2-7528cbf39197
7+
jinja: "{{sentence}}\nReplace the _ in the above sentence with the correct option:\
8+
\ \n- {{option1}}\n- {{option2}}\n|||\n{% if answer == '1' %} {{option1}} {%\
9+
\ else %} {{ option2 }} {% endif %}"
10+
metadata: !TemplateMetadata
11+
choices_in_prompt: true
12+
languages:
13+
- en
14+
metrics:
15+
- Accuracy
16+
original_task: true
17+
name: Replace
18+
reference: ''
19+
60ce5113-882f-4a9d-b21d-8d98b4644295: !Template
20+
answer_choices: '{{option1}} ||| {{option2}}'
21+
id: 60ce5113-882f-4a9d-b21d-8d98b4644295
22+
jinja: 'Fill in the _ in the below sentence:
23+
24+
{{sentence}}
25+
26+
27+
Choices:
28+
29+
- {{ option1 }}
30+
31+
- {{ option2 }}
32+
33+
34+
Answer: ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {%
35+
endif %}'
36+
metadata: !TemplateMetadata
37+
choices_in_prompt: true
38+
languages:
39+
- en
40+
metrics:
41+
- Accuracy
42+
original_task: true
43+
name: fill in the blank
44+
reference: ''
45+
8f0f6d33-25e2-4394-b1f0-49a2a54767aa: !Template
46+
answer_choices: True ||| False
47+
id: 8f0f6d33-25e2-4394-b1f0-49a2a54767aa
48+
jinja: 'The _ in the sentence below refers to {{option1}}. True or False?
49+
50+
{{sentence}}|||
51+
52+
{{answer_choices[answer|int - 1]}}'
53+
metadata: !TemplateMetadata
54+
choices_in_prompt: true
55+
languages:
56+
- en
57+
metrics:
58+
- Accuracy
59+
original_task: false
60+
name: True or False
61+
reference: ''
62+
90f9679e-7b6c-4ee7-a348-e905ed9aaf9e: !Template
63+
answer_choices: '{{ option1 }} ||| {{ option2 }}'
64+
id: 90f9679e-7b6c-4ee7-a348-e905ed9aaf9e
65+
jinja: '{{ sentence }} In the previous sentence, does _ refer to {{ option1 }}
66+
or {{ option2 }}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2
67+
}} {% endif %}'
68+
metadata: !TemplateMetadata
69+
choices_in_prompt: true
70+
languages:
71+
- en
72+
metrics:
73+
- Accuracy
74+
original_task: true
75+
name: does underscore refer to
76+
reference: ''
77+
cd40cf1f-bda2-4757-b1b5-f1a20a3f7202: !Template
78+
answer_choices: '{{option1}} ||| {{option2}}'
79+
id: cd40cf1f-bda2-4757-b1b5-f1a20a3f7202
80+
jinja: '{{sentence}}
81+
82+
What does the _ in the above sentence refer to? {{ option1 }} or {{ option2
83+
}}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {% endif
84+
%}'
85+
metadata: !TemplateMetadata
86+
choices_in_prompt: true
87+
languages:
88+
- en
89+
metrics:
90+
- Accuracy
91+
original_task: true
92+
name: underscore refer to
93+
reference: ''
94+
fc365d5d-bb5c-488c-93a0-4f90e6011c5d: !Template
95+
answer_choices: '{{option1}} ||| {{option2}}'
96+
id: fc365d5d-bb5c-488c-93a0-4f90e6011c5d
97+
jinja: 'In the sentence below, does the _ stand for {{answer_choices[0]}} or {{answer_choices[1]}}?
98+
99+
{{sentence}}|||
100+
101+
{{answer_choices[answer | int - 1]}}'
102+
metadata: !TemplateMetadata
103+
choices_in_prompt: true
104+
languages:
105+
- en
106+
metrics:
107+
- Accuracy
108+
original_task: true
109+
name: stand for
110+
reference: ''

‎promptsource/templates/Muennighoff/xwinograd/pt/templates.yaml

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
dataset: Muennighoff/xwinograd
2+
subset: en
3+
templates:
4+
38d31908-4aee-4545-aff2-7528cbf39197: !Template
5+
answer_choices: '{{option1}} ||| {{option2}}'
6+
id: 38d31908-4aee-4545-aff2-7528cbf39197
7+
jinja: "{{sentence}}\nReplace the _ in the above sentence with the correct option:\
8+
\ \n- {{option1}}\n- {{option2}}\n|||\n{% if answer == '1' %} {{option1}} {%\
9+
\ else %} {{ option2 }} {% endif %}"
10+
metadata: !TemplateMetadata
11+
choices_in_prompt: true
12+
languages:
13+
- en
14+
metrics:
15+
- Accuracy
16+
original_task: true
17+
name: Replace
18+
reference: ''
19+
60ce5113-882f-4a9d-b21d-8d98b4644295: !Template
20+
answer_choices: '{{option1}} ||| {{option2}}'
21+
id: 60ce5113-882f-4a9d-b21d-8d98b4644295
22+
jinja: 'Fill in the _ in the below sentence:
23+
24+
{{sentence}}
25+
26+
27+
Choices:
28+
29+
- {{ option1 }}
30+
31+
- {{ option2 }}
32+
33+
34+
Answer: ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {%
35+
endif %}'
36+
metadata: !TemplateMetadata
37+
choices_in_prompt: true
38+
languages:
39+
- en
40+
metrics:
41+
- Accuracy
42+
original_task: true
43+
name: fill in the blank
44+
reference: ''
45+
8f0f6d33-25e2-4394-b1f0-49a2a54767aa: !Template
46+
answer_choices: True ||| False
47+
id: 8f0f6d33-25e2-4394-b1f0-49a2a54767aa
48+
jinja: 'The _ in the sentence below refers to {{option1}}. True or False?
49+
50+
{{sentence}}|||
51+
52+
{{answer_choices[answer|int - 1]}}'
53+
metadata: !TemplateMetadata
54+
choices_in_prompt: true
55+
languages:
56+
- en
57+
metrics:
58+
- Accuracy
59+
original_task: false
60+
name: True or False
61+
reference: ''
62+
90f9679e-7b6c-4ee7-a348-e905ed9aaf9e: !Template
63+
answer_choices: '{{ option1 }} ||| {{ option2 }}'
64+
id: 90f9679e-7b6c-4ee7-a348-e905ed9aaf9e
65+
jinja: '{{ sentence }} In the previous sentence, does _ refer to {{ option1 }}
66+
or {{ option2 }}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2
67+
}} {% endif %}'
68+
metadata: !TemplateMetadata
69+
choices_in_prompt: true
70+
languages:
71+
- en
72+
metrics:
73+
- Accuracy
74+
original_task: true
75+
name: does underscore refer to
76+
reference: ''
77+
cd40cf1f-bda2-4757-b1b5-f1a20a3f7202: !Template
78+
answer_choices: '{{option1}} ||| {{option2}}'
79+
id: cd40cf1f-bda2-4757-b1b5-f1a20a3f7202
80+
jinja: '{{sentence}}
81+
82+
What does the _ in the above sentence refer to? {{ option1 }} or {{ option2
83+
}}? ||| {% if answer == ''1'' %} {{option1}} {% else %} {{ option2 }} {% endif
84+
%}'
85+
metadata: !TemplateMetadata
86+
choices_in_prompt: true
87+
languages:
88+
- en
89+
metrics:
90+
- Accuracy
91+
original_task: true
92+
name: underscore refer to
93+
reference: ''
94+
fc365d5d-bb5c-488c-93a0-4f90e6011c5d: !Template
95+
answer_choices: '{{option1}} ||| {{option2}}'
96+
id: fc365d5d-bb5c-488c-93a0-4f90e6011c5d
97+
jinja: 'In the sentence below, does the _ stand for {{answer_choices[0]}} or {{answer_choices[1]}}?
98+
99+
{{sentence}}|||
100+
101+
{{answer_choices[answer | int - 1]}}'
102+
metadata: !TemplateMetadata
103+
choices_in_prompt: true
104+
languages:
105+
- en
106+
metrics:
107+
- Accuracy
108+
original_task: true
109+
name: stand for
110+
reference: ''

0 commit comments

Comments
 (0)