-
-
Notifications
You must be signed in to change notification settings - Fork 715
Batch Trigger upgrades #1502
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Batch Trigger upgrades #1502
Changes from all commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
7ec44c2
WIP batch trigger v2
ericallam e9f2f8c
Fix for the DateField being one month out… getUTCMonth() is zero inde…
matt-aitken 5ee6b8e
Added a custom date range filter
matt-aitken b4ed220
Deal with closing the custom date range
matt-aitken bd14761
Child runs filter
matt-aitken 6097160
Fix for the clear button untoggling the child runs
matt-aitken 4aa5845
WIP batchTriggerV2
ericallam d69abd1
Finished removing rate limit from the webapp
ericallam a664859
Added an index TaskRun to make useRealtimeBatch performant
ericallam a0dfd2d
Renamed the period filter labels to be “Last X mins”
matt-aitken daec8f4
Denormalize background worker columns into TaskRun
ericallam aa82f8d
Use the runTags column on TaskRun
matt-aitken 976b275
Add TaskRun ("projectId", "id" DESC) index
matt-aitken 1674d33
Improved the v2 batch trigger endpoint to process items in parallel a…
ericallam e8cfe88
Added a runId filter, and WIP for batchId filter
matt-aitken b0c3c41
WIP triggerAll
ericallam 4b181fa
Add new batch methods for triggering multiple different tasks in a si…
ericallam fd18784
Disabled switch styling
matt-aitken 26980f0
Batch filtering, force child runs to show if filtering by batch/run
matt-aitken 492fb79
Added schedule ID filtering
matt-aitken 0060a64
Force child runs to show when filtering by scheduleId, for consistency
matt-aitken f1919b1
realtime: allow setting enabled: false on useApiClient
ericallam 1b34daf
Batches page
matt-aitken 2ebc4e5
Always complete batches, not only batchTriggerAndWait in deployed tasks
ericallam 658f400
Add batch.retrieve and allow filtering by batch in runs.list
ericallam ff65310
Renamed pending to “In progress”
matt-aitken 5f5cbe2
Tidied up the table a bit
matt-aitken e60b75b
Deal with old batches: “Legacy batch”
matt-aitken eac6aa7
Added the Batch to the run inspector
matt-aitken 4b2932a
Fixed the migration that created the new idempotency key index on Bat…
ericallam 4070c6a
Fixed the name of the idempotencyKeyExpiresAt option and now default …
ericallam f97d55e
Timezone fix: wrong month in Usage page dropdown
matt-aitken 78e3a56
The DateField now defaults to local time, but can be overriden to use…
matt-aitken f95569a
Don’t allow the task icon to get squished
matt-aitken b7599ed
BatchFilters removed unused imports
matt-aitken 99a3137
In the batch filtering, use `id` instead of `batchId` in the URL
matt-aitken 681e75a
BatchFilters: we don’t need a child tasks hidden input field
matt-aitken bc07b52
Creates some common filter components/functions
matt-aitken b822915
Fix for batchVersion check when filtering by batch status
matt-aitken 4c5fd2a
Add additional logging around telemetry and more attributes for trigg…
ericallam fc821ad
Show clear button for specific id filters
matt-aitken d5c1bda
Batch list: only allow environments that are part of this project
matt-aitken 69f3dca
Unnecessary optional chain
matt-aitken 153af67
Add JSDocs
ericallam File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
--- | ||
"@trigger.dev/sdk": patch | ||
"trigger.dev": patch | ||
"@trigger.dev/core": patch | ||
--- | ||
|
||
Added new batch.trigger and batch.triggerByTask methods that allows triggering multiple different tasks in a single batch: | ||
|
||
```ts | ||
import { batch } from '@trigger.dev/sdk/v3'; | ||
import type { myTask1, myTask2 } from './trigger/tasks'; | ||
|
||
// Somewhere in your backend code | ||
const response = await batch.trigger<typeof myTask1 | typeof myTask2>([ | ||
{ id: 'task1', payload: { foo: 'bar' } }, | ||
{ id: 'task2', payload: { baz: 'qux' } }, | ||
]); | ||
|
||
for (const run of response.runs) { | ||
if (run.ok) { | ||
console.log(run.output); | ||
} else { | ||
console.error(run.error); | ||
} | ||
} | ||
``` | ||
|
||
Or if you are inside of a task, you can use `triggerByTask`: | ||
|
||
```ts | ||
import { batch, task, runs } from '@trigger.dev/sdk/v3'; | ||
|
||
export const myParentTask = task({ | ||
id: 'myParentTask', | ||
run: async () => { | ||
const response = await batch.triggerByTask([ | ||
{ task: myTask1, payload: { foo: 'bar' } }, | ||
{ task: myTask2, payload: { baz: 'qux' } }, | ||
]); | ||
|
||
const run1 = await runs.retrieve(response.runs[0]); | ||
console.log(run1.output) // typed as { foo: string } | ||
|
||
const run2 = await runs.retrieve(response.runs[1]); | ||
console.log(run2.output) // typed as { baz: string } | ||
|
||
ericallam marked this conversation as resolved.
Show resolved
Hide resolved
|
||
const response2 = await batch.triggerByTaskAndWait([ | ||
{ task: myTask1, payload: { foo: 'bar' } }, | ||
{ task: myTask2, payload: { baz: 'qux' } }, | ||
]); | ||
|
||
if (response2.runs[0].ok) { | ||
console.log(response2.runs[0].output) // typed as { foo: string } | ||
} | ||
|
||
if (response2.runs[1].ok) { | ||
console.log(response2.runs[1].output) // typed as { baz: string } | ||
} | ||
} | ||
}); | ||
|
||
export const myTask1 = task({ | ||
id: 'myTask1', | ||
run: async () => { | ||
return { | ||
foo: 'bar' | ||
} | ||
} | ||
}); | ||
|
||
export const myTask2 = task({ | ||
id: 'myTask2', | ||
run: async () => { | ||
return { | ||
baz: 'qux' | ||
} | ||
} | ||
}); | ||
|
||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
--- | ||
"@trigger.dev/react-hooks": minor | ||
"@trigger.dev/sdk": minor | ||
"@trigger.dev/core": minor | ||
--- | ||
|
||
Improved Batch Triggering: | ||
|
||
- The new Batch Trigger endpoint is now asynchronous and supports up to 500 runs per request. | ||
- The new endpoint also supports triggering multiple different tasks in a single batch request (support in the SDK coming soon). | ||
- The existing `batchTrigger` method now supports the new endpoint, and shouldn't require any changes to your code. | ||
|
||
- Idempotency keys now expire after 24 hours, and you can customize the expiration time when creating a new key by using the `idempotencyKeyTTL` parameter: | ||
|
||
```ts | ||
await myTask.batchTrigger([{ payload: { foo: "bar" }}], { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" }) | ||
// Works for individual items as well: | ||
await myTask.batchTrigger([{ payload: { foo: "bar" }, options: { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" }}]) | ||
// And `trigger`: | ||
await myTask.trigger({ foo: "bar" }, { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" }); | ||
``` | ||
|
||
### Breaking Changes | ||
|
||
- We've removed the `idempotencyKey` option from `triggerAndWait` and `batchTriggerAndWait`, because it can lead to permanently frozen runs in deployed tasks. We're working on upgrading our entire system to support idempotency keys on these methods, and we'll re-add the option once that's complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
--- | ||
"@trigger.dev/react-hooks": patch | ||
"@trigger.dev/sdk": patch | ||
"@trigger.dev/core": patch | ||
--- | ||
|
||
Added ability to subscribe to a batch of runs using runs.subscribeToBatch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
This is the repo for Trigger.dev, a background jobs platform written in TypeScript. Our webapp at apps/webapp is a Remix 2.1 app that uses Node.js v20. Our SDK is an isomorphic TypeScript SDK at packages/trigger-sdk. Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code. Our tests are all vitest. We use prisma in internal-packages/database for our database interactions using PostgreSQL. For TypeScript, we usually use types over interfaces. We use zod a lot in packages/core and in the webapp. Avoid enums. Use strict mode. No default exports, use function declarations. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.