Skip to content

Add english prompts for GEM/wikilingua (18 languages) #765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 31, 2022

Conversation

haileyschoelkopf
Copy link

Added 5 English prompts for each of the 18 languages in GEM/wikilingua!

Will need others to write non-english prompts for this dataset if time permits.

@afaji
Copy link

afaji commented May 8, 2022

@haileyschoelkopf I can assist the Indonesian

@haileyschoelkopf
Copy link
Author

@afaji That would be great! Thank you

@afaji
Copy link

afaji commented May 10, 2022

https://github.com/haileyschoelkopf/promptsource/pull/1/files <- Indonesian prompt for wikilingua

@stephenbach stephenbach self-assigned this May 18, 2022
@stephenbach
Copy link
Member

Looks good! Should the article_summary_xx prompts indicate what the language should be?

@haileyschoelkopf
Copy link
Author

I could definitely make that change if you think it's better!

My idea was to have a prompt that didn't indicate the language (since these dataset splits all have the same source and target language) to see if it degraded performance when summary language wasn't explicitly specified, but that might not be desirable for the hackathon evals.

@stephenbach
Copy link
Member

Yes, I think we should change it (or remove if that makes the prompt redundant with others?) It's a potentially interesting comparison to make, but balanced against the overall criteria for prompts in promptsource, I think it's better to have the task explicitly stated. CCing @awebson to see if he agrees?

@haileyschoelkopf
Copy link
Author

Of course, makes sense! I've made the changes and explicitly stated language for the tldr_xx prompts as well.

Co-authored-by: Stephen Bach <[email protected]>
Copy link
Member

@stephenbach stephenbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks!

@stephenbach stephenbach merged commit 1b35aa0 into bigscience-workshop:eval-hackathon May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants