Skip to content

AI-powered marking #1248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5cd6ac6
feat: v1 of AI-generated comments
xxdydx Feb 2, 2025
853ba84
feat: added logging of inputs and outputs
xxdydx Feb 13, 2025
4c37d14
Update generate_ai_comments.ex
xxdydx Feb 21, 2025
8192b3d
feat: function to save outputs to database
xxdydx Mar 17, 2025
8a235b3
Format answers json before sending to LLM
EugeneOYZ1203n Mar 18, 2025
d384e06
Add LLM Prompt to question params when submitting assessment xml file
EugeneOYZ1203n Mar 18, 2025
98feac2
Add LLM Prompt to api response when grading view is open
EugeneOYZ1203n Mar 18, 2025
7716d57
feat: added llm_prompt from qn to raw_prompt
xxdydx Mar 19, 2025
df34dbd
feat: enabling/disabling of LLM feature by course level
xxdydx Mar 19, 2025
0a25fa8
feat: added llm_grading boolean field to course creation API
xxdydx Mar 19, 2025
2723f5a
feat: added api key storage in courses & edit api key/enable llm grading
xxdydx Mar 26, 2025
02f7ed1
feat: encryption for llm_api_key
xxdydx Apr 2, 2025
cb34984
feat: added final comment editing route
xxdydx Apr 5, 2025
09a7b09
feat: added logging of chosen comments
xxdydx Apr 6, 2025
ed44a7e
fix: bugs when certain fields were missing
xxdydx Apr 6, 2025
3715368
feat: updated tests
xxdydx Apr 6, 2025
5bfe276
formatting
xxdydx Apr 6, 2025
c27b93b
Merge branch 'master' into feat/add-AI-generated-comments-grading
xxdydx Apr 6, 2025
17884fd
fix: error handling when calling openai API
xxdydx Apr 6, 2025
f91cc92
fix: credo issues
xxdydx Apr 9, 2025
81e5bf7
formatting
xxdydx Apr 9, 2025
8f8b93a
Merge branch 'master' into feat/add-AI-generated-comments-grading
RichDom2185 Jun 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,6 @@ erl_crash.dump

# Generated lexer
/src/source_lexer.erl

# Ignore log files
/log
95 changes: 95 additions & 0 deletions lib/cadet/ai_comments.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
defmodule Cadet.AIComments do
@moduledoc """
Handles operations related to AI comments, including creation, updates, and retrieval.
"""

import Ecto.Query
alias Cadet.Repo
alias Cadet.AIComments.AIComment

@doc """
Creates a new AI comment log entry.
"""
def create_ai_comment(attrs \\ %{}) do
%AIComment{}
|> AIComment.changeset(attrs)
|> Repo.insert()
end

@doc """
Gets an AI comment by ID.
"""
def get_ai_comment!(id), do: Repo.get!(AIComment, id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why we use get! instead of get


@doc """
Retrieves an AI comment for a specific submission and question.
Returns `nil` if no comment exists.
"""
def get_ai_comments_for_submission(submission_id, question_id) do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming implies you are getting all AI comments. Also what is the use case for getting only one of the comments?

Repo.one(
from(c in AIComment,
where: c.submission_id == ^submission_id and c.question_id == ^question_id
)
)
end

@doc """
Retrieves the latest AI comment for a specific submission and question.
Returns `nil` if no comment exists.
"""
def get_latest_ai_comment(submission_id, question_id) do
Repo.one(
from(c in AIComment,
where: c.submission_id == ^submission_id and c.question_id == ^question_id,
order_by: [desc: c.inserted_at],
limit: 1
)
)
end

@doc """
Updates the final comment for a specific submission and question.
Returns the most recent comment entry for that submission/question.
"""
def update_final_comment(submission_id, question_id, final_comment) do
comment = get_latest_ai_comment(submission_id, question_id)

case comment do
nil ->
{:error, :not_found}

_ ->
comment
|> AIComment.changeset(%{final_comment: final_comment})
|> Repo.update()
end
end

@doc """
Updates an existing AI comment with new attributes.
"""
def update_ai_comment(id, attrs) do
id
|> get_ai_comment!()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could raise an error which isn't handled (id not found in DB)

|> AIComment.changeset(attrs)
|> Repo.update()
end

@doc """
Updates the chosen comments for a specific submission and question.
Accepts an array of comments and replaces the existing array in the database.
"""
def update_chosen_comments(submission_id, question_id, new_comments) do
comment = get_latest_ai_comment(submission_id, question_id)

case comment do
nil ->
{:error, :not_found}

_ ->
comment
|> AIComment.changeset(%{comment_chosen: new_comments})
|> Repo.update()
end
end
end
36 changes: 36 additions & 0 deletions lib/cadet/ai_comments/ai_comment.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
defmodule Cadet.AIComments.AIComment do
@moduledoc """
Defines the schema and changeset for AI comments.
"""

use Ecto.Schema
import Ecto.Changeset

schema "ai_comment_logs" do
field(:submission_id, :integer)
field(:question_id, :integer)
Comment on lines +10 to +11
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these are FKs, you need to specify that in the schema

field(:raw_prompt, :string)
field(:answers_json, :string)
field(:response, :string)
field(:error, :string)
field(:comment_chosen, {:array, :string})
field(:final_comment, :string)

timestamps()
end

def changeset(ai_comment, attrs) do
ai_comment
|> cast(attrs, [
:submission_id,
:question_id,
:raw_prompt,
:answers_json,
:response,
:error,
:comment_chosen,
:final_comment
])
|> validate_required([:submission_id, :question_id, :raw_prompt, :answers_json])
end
end
11 changes: 9 additions & 2 deletions lib/cadet/assessments/assessments.ex
Original file line number Diff line number Diff line change
Expand Up @@ -2286,8 +2286,8 @@ defmodule Cadet.Assessments do
@spec get_answers_in_submission(integer() | String.t()) ::
{:ok, {[Answer.t()], Assessment.t()}}
| {:error, {:bad_request, String.t()}}
def get_answers_in_submission(id) when is_ecto_id(id) do
answer_query =
def get_answers_in_submission(id, question_id \\ nil) when is_ecto_id(id) do
base_query =
Answer
|> where(submission_id: ^id)
|> join(:inner, [a], q in assoc(a, :question))
Expand All @@ -2309,6 +2309,13 @@ defmodule Cadet.Assessments do
{s, student: {st, user: u}, team: {t, team_members: {tm, student: {tms, user: tmu}}}}
)

answer_query =
if is_nil(question_id) do
base_query
else
base_query |> where(question_id: ^question_id)
end

Comment on lines +2312 to +2318
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use case instead

answers =
answer_query
|> Repo.all()
Expand Down
3 changes: 2 additions & 1 deletion lib/cadet/assessments/question_types/programming_question.ex
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,14 @@ defmodule Cadet.Assessments.QuestionTypes.ProgrammingQuestion do
field(:template, :string)
field(:postpend, :string, default: "")
field(:solution, :string)
field(:llm_prompt, :string)
embeds_many(:public, Testcase)
embeds_many(:opaque, Testcase)
embeds_many(:secret, Testcase)
end

@required_fields ~w(content template)a
@optional_fields ~w(solution prepend postpend)a
@optional_fields ~w(solution prepend postpend llm_prompt)a

def changeset(question, params \\ %{}) do
question
Expand Down
54 changes: 52 additions & 2 deletions lib/cadet/courses/course.ex
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ defmodule Cadet.Courses.Course do
enable_achievements: boolean(),
enable_sourcecast: boolean(),
enable_stories: boolean(),
enable_llm_grading: boolean(),
llm_api_key: String.t() | nil,
source_chapter: integer(),
source_variant: String.t(),
module_help_text: String.t(),
Expand All @@ -28,6 +30,8 @@ defmodule Cadet.Courses.Course do
field(:enable_achievements, :boolean, default: true)
field(:enable_sourcecast, :boolean, default: true)
field(:enable_stories, :boolean, default: false)
field(:enable_llm_grading, :boolean)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be defaulted to false

field(:llm_api_key, :string)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I understanding this right that you are going to have separate API keys for each course...? Why is it not just set in the env file.

field(:source_chapter, :integer)
field(:source_variant, :string)
field(:module_help_text, :string)
Expand All @@ -42,13 +46,59 @@ defmodule Cadet.Courses.Course do

@required_fields ~w(course_name viewable enable_game
enable_achievements enable_sourcecast enable_stories source_chapter source_variant)a
@optional_fields ~w(course_short_name module_help_text)a

@optional_fields ~w(course_short_name module_help_text enable_llm_grading llm_api_key)a

@spec changeset(
{map(), map()}
| %{
:__struct__ => atom() | %{:__changeset__ => map(), optional(any()) => any()},
optional(atom()) => any()
},
%{optional(:__struct__) => none(), optional(atom() | binary()) => any()}
) :: Ecto.Changeset.t()
def changeset(course, params) do
course
|> cast(params, @required_fields ++ @optional_fields)
|> validate_required(@required_fields)
|> validate_sublanguage_combination(params)
|> put_encrypted_llm_api_key()
end

def put_encrypted_llm_api_key(changeset) do
if llm_api_key = get_change(changeset, :llm_api_key) do
if is_binary(llm_api_key) and llm_api_key != "" do
secret = Application.get_env(:openai, :encryption_key)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding this to configs if this is something new. i.e. a comment for it in configs


if is_binary(secret) and byte_size(secret) >= 16 do
# Use first 16 bytes for AES-128, 24 for AES-192, or 32 for AES-256
key = binary_part(secret, 0, min(32, byte_size(secret)))
# Use AES in GCM mode for encryption
iv = :crypto.strong_rand_bytes(16)

{ciphertext, tag} =
:crypto.crypto_one_time_aead(
:aes_gcm,
key,
iv,
llm_api_key,
"",
true
)

# Store both the IV, ciphertext and tag
encrypted = iv <> tag <> ciphertext
put_change(changeset, :llm_api_key, Base.encode64(encrypted))
else
add_error(changeset, :llm_api_key, "encryption key not configured properly")
end
else
# If empty string or nil is provided, don't encrypt but don't add error
changeset
end
else
# The key is not being changed, so we need to preserve the existing value
put_change(changeset, :llm_api_key, changeset.data.llm_api_key)
end
end

# Validates combination of Source chapter and variant
Expand Down
3 changes: 2 additions & 1 deletion lib/cadet/jobs/xml_parser.ex
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,8 @@ defmodule Cadet.Updater.XMLParser do
prepend: ~x"./SNIPPET/PREPEND/text()" |> transform_by(&process_charlist/1),
template: ~x"./SNIPPET/TEMPLATE/text()" |> transform_by(&process_charlist/1),
postpend: ~x"./SNIPPET/POSTPEND/text()" |> transform_by(&process_charlist/1),
solution: ~x"./SNIPPET/SOLUTION/text()" |> transform_by(&process_charlist/1)
solution: ~x"./SNIPPET/SOLUTION/text()" |> transform_by(&process_charlist/1),
llm_prompt: ~x"./LLM_GRADING_PROMPT/text()" |> transform_by(&process_charlist/1)
),
entity
|> xmap(
Expand Down
2 changes: 2 additions & 0 deletions lib/cadet_web/admin_controllers/admin_courses_controller.ex
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ defmodule CadetWeb.AdminCoursesController do
enable_achievements(:body, :boolean, "Enable achievements")
enable_sourcecast(:body, :boolean, "Enable sourcecast")
enable_stories(:body, :boolean, "Enable stories")
enable_llm_grading(:body, :boolean, "Enable LLM grading")
llm_api_key(:body, :string, "OpenAI API key for this course")
sublanguage(:body, Schema.ref(:AdminSublanguage), "sublanguage object")
module_help_text(:body, :string, "Module help text")
end
Expand Down
6 changes: 6 additions & 0 deletions lib/cadet_web/controllers/courses_controller.ex
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ defmodule CadetWeb.CoursesController do
enable_achievements(:body, :boolean, "Enable achievements", required: true)
enable_sourcecast(:body, :boolean, "Enable sourcecast", required: true)
enable_stories(:body, :boolean, "Enable stories", required: true)
enable_llm_grading(:body, :boolean, "Enable LLM grading", required: false)
llm_api_key(:body, :string, "OpenAI API key for this course", required: false)
source_chapter(:body, :number, "Default source chapter", required: true)

source_variant(:body, Schema.ref(:SourceVariant), "Default source variant name",
Expand Down Expand Up @@ -97,6 +99,8 @@ defmodule CadetWeb.CoursesController do
enable_achievements(:boolean, "Enable achievements", required: true)
enable_sourcecast(:boolean, "Enable sourcecast", required: true)
enable_stories(:boolean, "Enable stories", required: true)
enable_llm_grading(:boolean, "Enable LLM grading", required: false)
llm_api_key(:string, "OpenAI API key for this course", required: false)
source_chapter(:integer, "Source Chapter number from 1 to 4", required: true)
source_variant(Schema.ref(:SourceVariant), "Source Variant name", required: true)
module_help_text(:string, "Module help text", required: true)
Expand All @@ -111,6 +115,8 @@ defmodule CadetWeb.CoursesController do
enable_achievements: true,
enable_sourcecast: true,
enable_stories: false,
enable_llm_grading: false,
llm_api_key: "sk-1234567890",
source_chapter: 1,
source_variant: "default",
module_help_text: "Help text",
Expand Down
Loading
Loading