Skip to content

Add prompts for schema_guided_dstc8 response generation #756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

kasnerz
Copy link

@kasnerz kasnerz commented Apr 28, 2022

Added 5 prompts for chatbot response generation

@kasnerz kasnerz changed the title Add templates for schema_guided_dstc8 response generation Add prompts for schema_guided_dstc8 response generation Apr 28, 2022
@stephenbach stephenbach self-assigned this May 12, 2022
@stephenbach
Copy link
Member

Thanks! Looks good! Just a couple of quick questions and thoughts:

  • Should these be marked as "original task?"
  • There are some new lines at the end of the targets that are added in some the prompt templates. These won't matter for methods that don't tokenize newlines separately, but probably best just to remove them. I assume you don't want to score methods based on that?

@kasnerz
Copy link
Author

kasnerz commented May 18, 2022

  • No, I do not think the prompts should be marked as "original task". The dataset was developed with dialogue state tracking for the DSTC8 challenge in mind, which is quite different from the (stateless) response generation.
  • The newlines are not intentional, it is probably just an artifact from the web interface. I tried to remove them in the YAML file, please check if I did it correctly. I also noticed the same pattern is present in other prompts (e.g. MetaWOZ which I was using as an inspiration).

@stephenbach
Copy link
Member

Looks great, thanks! Can you just merge in eval-hackathon again so that the tests run properly? Thanks.

@kasnerz
Copy link
Author

kasnerz commented May 24, 2022

@stephenbach Done

Copy link
Member

@stephenbach stephenbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thank you!

@stephenbach stephenbach merged commit 9bd725a into bigscience-workshop:eval-hackathon May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants