Skip to content

Xcopa #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 16, 2022
Merged

Xcopa #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8206ffc
add EN prompts for xcopa
Jul 15, 2022
6af75cb
update ids (2/5)
Jul 15, 2022
1d4d0cc
regenerate ids (4/6)
Jul 15, 2022
e020740
regenerate ids (5/6)
Jul 15, 2022
8511070
regenerate ids (6/6)
Jul 15, 2022
cc65cf3
fix typo
Jul 15, 2022
670e6ad
fix last keyerror?
Jul 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 211 additions & 0 deletions promptsource/templates/xcopa/id/templates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
dataset: xcopa
subset: id
templates:
1a87b487-1570-4873-aed9-b84d2fc0476c: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 1a87b487-1570-4873-aed9-b84d2fc0476c
jinja: "{{ premise }} \n\nI am hesitating between two options. Help me choose\
\ the more likely {% if question == \"cause\" %} cause: {% else %} effect: {%\
\ endif %}\n- {{choice1}}\n- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label]\
\ }}{%endif%}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: i_am_hesitating
reference: ''
336c4c72-40e3-4122-881e-8cd7a1881eec: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 336c4c72-40e3-4122-881e-8cd7a1881eec
jinja: "{% if question == \"cause\" %} \n{{ premise }} Why? \"{{ answer_choices[0]\
\ }}\" or \"{{ answer_choices[1] }}\"? ||| {% if label != -1 %}{{ answer_choices[label]\
\ }}{%endif%}\n{% endif %}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: "\u2026why? C1 or C2"
reference: ''
482f0b87-e748-4e98-8cc8-a23386bc50c3: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 482f0b87-e748-4e98-8cc8-a23386bc50c3
jinja: "{{ premise }} \n\nWhat's the best option?\n- {{choice1}}\n- {{choice2}}\n\
\nWe are looking for {% if question == \"cause\" %} a cause {% else %} an effect\
\ {% endif %}\n||| {% if label != -1 %}{{answer_choices[label]}}{%endif%}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: best_option
reference: ''
4a0640a5-c378-422d-879b-7490bc500c8a: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 4a0640a5-c378-422d-879b-7490bc500c8a
jinja: '{{ premise }} {% if question == "cause" %} because... {% else %} so...
{% endif %}

Choose between:

- {{choice1}}

- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label] }}{%endif%}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: choose
reference: ''
78e28a66-a84c-442c-9bf7-44aa49450412: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 78e28a66-a84c-442c-9bf7-44aa49450412
jinja: '{{ premise }} {% if question == "cause" %} This happened because... {%
else %} As a consequence... {% endif %}

Help me pick the more plausible option:

- {{choice1}}

- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label] }}{%endif%}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: plausible_alternatives
reference: ''
7c0b578c-214f-4dc9-a9b4-252d91691cb0: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 7c0b578c-214f-4dc9-a9b4-252d91691cb0
jinja: "{% if question == \"effect\" %} \n{{ premise }} As a result, \"{{ answer_choices[0]\
\ }}\" or \"{{ answer_choices[1] }}\"? ||| {% if label != -1 %}{{ answer_choices[label]\
\ }}{%endif%}\n{% endif %}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: "\u2026As a result, C1 or C2?"
reference: ''
94b5be71-c989-4a62-96d9-a7cb042e83c7: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: 94b5be71-c989-4a62-96d9-a7cb042e83c7
jinja: 'Exercise: choose the most plausible alternative.


{{ premise }} {% if question == "cause" %} because... {% else %} so... {% endif
%}

- {{choice1}}

- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label] }}{%endif%}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: exercise
reference: ''
b308f6ce-673c-44c1-b84d-95a3045229ea: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: b308f6ce-673c-44c1-b84d-95a3045229ea
jinja: '"{{ answer_choices[0] }}" or "{{ answer_choices[1] }}"? {{ premise }}
{% if question == "cause" %} because {% else %} so {% endif %} ||| {% if label
!= -1 %}{{ answer_choices[label] }}{% endif %}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: "C1 or C2? premise, so/because\u2026"
reference: "Adapted from Perez et al. 2021 and Schick & Sch\xFCtz 2021."
cf78cf75-90cc-4fe2-8b78-2bf64c9520b4: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: cf78cf75-90cc-4fe2-8b78-2bf64c9520b4
jinja: '{{ premise }}


Select the most plausible {% if question == "cause" %} cause: {% else %} effect:
{% endif %}

- {{choice1}}

- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label] }}{%endif%}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: cause_effect
reference: ''
d8263afb-215f-43c4-83b8-c85744144fdb: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: d8263afb-215f-43c4-83b8-c85744144fdb
jinja: "{% if question == \"cause\" %} \n{{ premise }} Which may be caused by\
\ \"{{ answer_choices[0] }}\" or \"{{ answer_choices[1] }}\"? ||| {% if label\
\ != -1 %}{{ answer_choices[label] }}{%endif%}\n{% endif %}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: "\u2026which may be caused by"
reference: ''
eaddf2e0-ead4-456b-8e81-00bdcde8c7b0: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: eaddf2e0-ead4-456b-8e81-00bdcde8c7b0
jinja: "{% if question == \"effect\" %} \n{{ premise }} What could happen next,\
\ \"{{ answer_choices[0] }}\" or \"{{ answer_choices[1] }}\"? ||| {% if label\
\ != -1 %}{{ answer_choices[label] }}{%endif%}\n{% endif %}"
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: "\u2026What could happen next, C1 or C2?"
reference: ''
ebd4242a-14f2-4aed-a183-dc37a18dfe4b: !Template
answer_choices: '{{choice1}} ||| {{choice2}}'
id: ebd4242a-14f2-4aed-a183-dc37a18dfe4b
jinja: 'Pick the more likely continuation to the following sentence:

{{ premise }} {% if question == "cause" %} as a result of: {% else %} as a consequence:
{% endif %}

- {{choice1}}

- {{choice2}} ||| {% if label != -1 %}{{ answer_choices[label] }}{%endif%}'
metadata: !TemplateMetadata
choices_in_prompt: true
languages:
- en
metrics:
- Accuracy
original_task: true
name: more likely
reference: ''
Loading