Enhance Token Counting to Include Chat History and System Instructions #64
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… and history usage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This pull request introduces enhancements to the token counting functionalities within our chat models. The updates ensure that token counting reflects the entire context of a chat session, including chat history and system instructions, aligning our implementation with the official Gemini API guidelines.
Changes
Chat Session Token Counting: Added a new function in the
ChatSessionclass to calculate the total token count that includes the history of the chat session. This modification allows us to capture the multi-turn nature of conversations, as described in the Gemini API documentation:Multi-turn Tokens Documentation.
System Instructions Integration: Updated the
GenerativeModelclass's token counting method to factor in system instructions. This ensures that our token counting is comprehensive and covers all elements that could potentially consume tokens as outlined here:System Instructions and Tools Documentation.
Motivation
The necessity for these updates was evident from the discussions in ticket #58, where it was highlighted that the current token counting mechanism did not account for the comprehensive context used by the Gemini API, leading to discrepancies in token usage calculations.
These enhancements ensure that our models are fully compliant with the guidelines and functionalities supported by the Gemini API, providing a more accurate and reliable count of token usage across different scenarios.
I look forward to your feedback on these enhancements.