Skip to content

Batch Trigger upgrades #1502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Nov 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
7ec44c2
WIP batch trigger v2
ericallam Nov 22, 2024
e9f2f8c
Fix for the DateField being one month out… getUTCMonth() is zero inde…
matt-aitken Nov 20, 2024
5ee6b8e
Added a custom date range filter
matt-aitken Nov 21, 2024
b4ed220
Deal with closing the custom date range
matt-aitken Nov 21, 2024
bd14761
Child runs filter
matt-aitken Nov 21, 2024
6097160
Fix for the clear button untoggling the child runs
matt-aitken Nov 21, 2024
4aa5845
WIP batchTriggerV2
ericallam Nov 25, 2024
d69abd1
Finished removing rate limit from the webapp
ericallam Nov 25, 2024
a664859
Added an index TaskRun to make useRealtimeBatch performant
ericallam Nov 26, 2024
a0dfd2d
Renamed the period filter labels to be “Last X mins”
matt-aitken Nov 25, 2024
daec8f4
Denormalize background worker columns into TaskRun
ericallam Nov 26, 2024
aa82f8d
Use the runTags column on TaskRun
matt-aitken Nov 25, 2024
976b275
Add TaskRun ("projectId", "id" DESC) index
matt-aitken Nov 26, 2024
1674d33
Improved the v2 batch trigger endpoint to process items in parallel a…
ericallam Nov 26, 2024
e8cfe88
Added a runId filter, and WIP for batchId filter
matt-aitken Nov 26, 2024
b0c3c41
WIP triggerAll
ericallam Nov 26, 2024
4b181fa
Add new batch methods for triggering multiple different tasks in a si…
ericallam Nov 27, 2024
fd18784
Disabled switch styling
matt-aitken Nov 26, 2024
26980f0
Batch filtering, force child runs to show if filtering by batch/run
matt-aitken Nov 26, 2024
492fb79
Added schedule ID filtering
matt-aitken Nov 26, 2024
0060a64
Force child runs to show when filtering by scheduleId, for consistency
matt-aitken Nov 26, 2024
f1919b1
realtime: allow setting enabled: false on useApiClient
ericallam Nov 27, 2024
1b34daf
Batches page
matt-aitken Nov 27, 2024
2ebc4e5
Always complete batches, not only batchTriggerAndWait in deployed tasks
ericallam Nov 27, 2024
658f400
Add batch.retrieve and allow filtering by batch in runs.list
ericallam Nov 27, 2024
ff65310
Renamed pending to “In progress”
matt-aitken Nov 27, 2024
5f5cbe2
Tidied up the table a bit
matt-aitken Nov 27, 2024
e60b75b
Deal with old batches: “Legacy batch”
matt-aitken Nov 27, 2024
eac6aa7
Added the Batch to the run inspector
matt-aitken Nov 27, 2024
4b2932a
Fixed the migration that created the new idempotency key index on Bat…
ericallam Nov 27, 2024
4070c6a
Fixed the name of the idempotencyKeyExpiresAt option and now default …
ericallam Nov 27, 2024
f97d55e
Timezone fix: wrong month in Usage page dropdown
matt-aitken Nov 27, 2024
78e3a56
The DateField now defaults to local time, but can be overriden to use…
matt-aitken Nov 27, 2024
f95569a
Don’t allow the task icon to get squished
matt-aitken Nov 27, 2024
b7599ed
BatchFilters removed unused imports
matt-aitken Nov 27, 2024
99a3137
In the batch filtering, use `id` instead of `batchId` in the URL
matt-aitken Nov 27, 2024
681e75a
BatchFilters: we don’t need a child tasks hidden input field
matt-aitken Nov 27, 2024
bc07b52
Creates some common filter components/functions
matt-aitken Nov 27, 2024
b822915
Fix for batchVersion check when filtering by batch status
matt-aitken Nov 27, 2024
4c5fd2a
Add additional logging around telemetry and more attributes for trigg…
ericallam Nov 27, 2024
fc821ad
Show clear button for specific id filters
matt-aitken Nov 27, 2024
d5c1bda
Batch list: only allow environments that are part of this project
matt-aitken Nov 27, 2024
69f3dca
Unnecessary optional chain
matt-aitken Nov 27, 2024
153af67
Add JSDocs
ericallam Nov 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .changeset/perfect-onions-call.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
"@trigger.dev/sdk": patch
"trigger.dev": patch
"@trigger.dev/core": patch
---

Added new batch.trigger and batch.triggerByTask methods that allows triggering multiple different tasks in a single batch:

```ts
import { batch } from '@trigger.dev/sdk/v3';
import type { myTask1, myTask2 } from './trigger/tasks';

// Somewhere in your backend code
const response = await batch.trigger<typeof myTask1 | typeof myTask2>([
{ id: 'task1', payload: { foo: 'bar' } },
{ id: 'task2', payload: { baz: 'qux' } },
]);

for (const run of response.runs) {
if (run.ok) {
console.log(run.output);
} else {
console.error(run.error);
}
}
```

Or if you are inside of a task, you can use `triggerByTask`:

```ts
import { batch, task, runs } from '@trigger.dev/sdk/v3';

export const myParentTask = task({
id: 'myParentTask',
run: async () => {
const response = await batch.triggerByTask([
{ task: myTask1, payload: { foo: 'bar' } },
{ task: myTask2, payload: { baz: 'qux' } },
]);

const run1 = await runs.retrieve(response.runs[0]);
console.log(run1.output) // typed as { foo: string }

const run2 = await runs.retrieve(response.runs[1]);
console.log(run2.output) // typed as { baz: string }

const response2 = await batch.triggerByTaskAndWait([
{ task: myTask1, payload: { foo: 'bar' } },
{ task: myTask2, payload: { baz: 'qux' } },
]);

if (response2.runs[0].ok) {
console.log(response2.runs[0].output) // typed as { foo: string }
}

if (response2.runs[1].ok) {
console.log(response2.runs[1].output) // typed as { baz: string }
}
}
});

export const myTask1 = task({
id: 'myTask1',
run: async () => {
return {
foo: 'bar'
}
}
});

export const myTask2 = task({
id: 'myTask2',
run: async () => {
return {
baz: 'qux'
}
}
});

```
25 changes: 25 additions & 0 deletions .changeset/ten-pans-itch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
"@trigger.dev/react-hooks": minor
"@trigger.dev/sdk": minor
"@trigger.dev/core": minor
---

Improved Batch Triggering:

- The new Batch Trigger endpoint is now asynchronous and supports up to 500 runs per request.
- The new endpoint also supports triggering multiple different tasks in a single batch request (support in the SDK coming soon).
- The existing `batchTrigger` method now supports the new endpoint, and shouldn't require any changes to your code.

- Idempotency keys now expire after 24 hours, and you can customize the expiration time when creating a new key by using the `idempotencyKeyTTL` parameter:

```ts
await myTask.batchTrigger([{ payload: { foo: "bar" }}], { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" })
// Works for individual items as well:
await myTask.batchTrigger([{ payload: { foo: "bar" }, options: { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" }}])
// And `trigger`:
await myTask.trigger({ foo: "bar" }, { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" });
```

### Breaking Changes

- We've removed the `idempotencyKey` option from `triggerAndWait` and `batchTriggerAndWait`, because it can lead to permanently frozen runs in deployed tasks. We're working on upgrading our entire system to support idempotency keys on these methods, and we'll re-add the option once that's complete.
7 changes: 7 additions & 0 deletions .changeset/wild-needles-hunt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@trigger.dev/react-hooks": patch
"@trigger.dev/sdk": patch
"@trigger.dev/core": patch
---

Added ability to subscribe to a batch of runs using runs.subscribeToBatch
1 change: 1 addition & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is the repo for Trigger.dev, a background jobs platform written in TypeScript. Our webapp at apps/webapp is a Remix 2.1 app that uses Node.js v20. Our SDK is an isomorphic TypeScript SDK at packages/trigger-sdk. Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code. Our tests are all vitest. We use prisma in internal-packages/database for our database interactions using PostgreSQL. For TypeScript, we usually use types over interfaces. We use zod a lot in packages/core and in the webapp. Avoid enums. Use strict mode. No default exports, use function declarations.
9 changes: 9 additions & 0 deletions apps/webapp/app/components/navigation/SideMenu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
RectangleStackIcon,
ServerStackIcon,
ShieldCheckIcon,
Squares2X2Icon,
} from "@heroicons/react/20/solid";
import { UserGroupIcon, UserPlusIcon } from "@heroicons/react/24/solid";
import { useNavigation } from "@remix-run/react";
Expand Down Expand Up @@ -45,6 +46,7 @@ import {
projectSetupPath,
projectTriggersPath,
v3ApiKeysPath,
v3BatchesPath,
v3BillingPath,
v3ConcurrencyPath,
v3DeploymentsPath,
Expand Down Expand Up @@ -475,6 +477,13 @@ function V3ProjectSideMenu({
activeIconColor="text-teal-500"
to={v3RunsPath(organization, project)}
/>
<SideMenuItem
name="Batches"
icon={Squares2X2Icon}
activeIconColor="text-blue-500"
to={v3BatchesPath(organization, project)}
data-action="batches"
/>
<SideMenuItem
name="Test"
icon={BeakerIcon}
Expand Down
43 changes: 29 additions & 14 deletions apps/webapp/app/components/primitives/DateField.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { BellAlertIcon } from "@heroicons/react/20/solid";
import { BellAlertIcon, XMarkIcon } from "@heroicons/react/20/solid";
import { CalendarDateTime, createCalendar } from "@internationalized/date";
import { useDateField, useDateSegment } from "@react-aria/datepicker";
import type { DateFieldState, DateSegment } from "@react-stately/datepicker";
Expand All @@ -12,7 +12,7 @@ const variants = {
small: {
fieldStyles: "h-5 text-sm rounded-sm px-0.5",
nowButtonVariant: "tertiary/small" as const,
clearButtonVariant: "minimal/small" as const,
clearButtonVariant: "tertiary/small" as const,
},
medium: {
fieldStyles: "h-7 text-base rounded px-1",
Expand All @@ -35,9 +35,12 @@ type DateFieldProps = {
showNowButton?: boolean;
showClearButton?: boolean;
onValueChange?: (value: Date | undefined) => void;
utc?: boolean;
variant?: Variant;
};

const deviceTimezone = Intl.DateTimeFormat().resolvedOptions().timeZone;

export function DateField({
label,
defaultValue,
Expand All @@ -50,22 +53,23 @@ export function DateField({
showGuide = false,
showNowButton = false,
showClearButton = false,
utc = false,
variant = "small",
}: DateFieldProps) {
const [value, setValue] = useState<undefined | CalendarDateTime>(
utcDateToCalendarDate(defaultValue)
utc ? utcDateToCalendarDate(defaultValue) : dateToCalendarDate(defaultValue)
);

const state = useDateFieldState({
value: value,
onChange: (value) => {
if (value) {
setValue(value);
onValueChange?.(value.toDate("utc"));
onValueChange?.(value.toDate(utc ? "utc" : deviceTimezone));
}
},
minValue: utcDateToCalendarDate(minValue),
maxValue: utcDateToCalendarDate(maxValue),
minValue: utc ? utcDateToCalendarDate(minValue) : dateToCalendarDate(minValue),
maxValue: utc ? utcDateToCalendarDate(maxValue) : dateToCalendarDate(maxValue),
shouldForceLeadingZeros: true,
granularity,
locale: "en-US",
Expand All @@ -78,7 +82,9 @@ export function DateField({
useEffect(() => {
if (state.value === undefined && defaultValue === undefined) return;

const calendarDate = utcDateToCalendarDate(defaultValue);
const calendarDate = utc
? utcDateToCalendarDate(defaultValue)
: dateToCalendarDate(defaultValue);
//unchanged
if (state.value?.toDate("utc").getTime() === defaultValue?.getTime()) {
return;
Expand Down Expand Up @@ -134,23 +140,19 @@ export function DateField({
<Button
type="button"
variant={variants[variant].nowButtonVariant}
LeadingIcon={BellAlertIcon}
leadingIconClassName="text-text-dimmed group-hover:text-text-bright"
onClick={() => {
const now = new Date();
setValue(utcDateToCalendarDate(new Date()));
setValue(utc ? utcDateToCalendarDate(now) : dateToCalendarDate(now));
onValueChange?.(now);
}}
>
<span className="text-text-dimmed transition group-hover:text-text-bright">Now</span>
Now
</Button>
)}
{showClearButton && (
<Button
type="button"
variant={variants[variant].clearButtonVariant}
LeadingIcon={"close"}
leadingIconClassName="-mr-2"
onClick={() => {
setValue(undefined);
onValueChange?.(undefined);
Expand Down Expand Up @@ -181,7 +183,7 @@ function utcDateToCalendarDate(date?: Date) {
return date
? new CalendarDateTime(
date.getUTCFullYear(),
date.getUTCMonth(),
date.getUTCMonth() + 1,
date.getUTCDate(),
date.getUTCHours(),
date.getUTCMinutes(),
Expand All @@ -190,6 +192,19 @@ function utcDateToCalendarDate(date?: Date) {
: undefined;
}

function dateToCalendarDate(date?: Date) {
return date
? new CalendarDateTime(
date.getFullYear(),
date.getMonth() + 1,
date.getDate(),
date.getHours(),
date.getMinutes(),
date.getSeconds()
)
: undefined;
}

type DateSegmentProps = {
segment: DateSegment;
state: DateFieldState;
Expand Down
4 changes: 2 additions & 2 deletions apps/webapp/app/components/primitives/Select.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ export interface SelectItemProps extends Ariakit.SelectItemProps {
}

const selectItemClasses =
"group cursor-pointer px-1 pt-1 text-sm text-text-dimmed focus-custom last:pb-1";
"group cursor-pointer px-1 pt-1 text-2sm text-text-dimmed focus-custom last:pb-1";

export function SelectItem({
icon,
Expand Down Expand Up @@ -613,7 +613,7 @@ export function SelectPopover({
"z-50 flex flex-col overflow-clip rounded border border-charcoal-700 bg-background-bright shadow-md outline-none animate-in fade-in-40",
"min-w-[max(180px,calc(var(--popover-anchor-width)+0.5rem))]",
"max-w-[min(480px,var(--popover-available-width))]",
"max-h-[min(520px,var(--popover-available-height))]",
"max-h-[min(600px,var(--popover-available-height))]",
"origin-[var(--popover-transform-origin)]",
className
)}
Expand Down
4 changes: 2 additions & 2 deletions apps/webapp/app/components/primitives/Switch.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ const variations = {
},
small: {
container:
"flex items-center h-[1.5rem] gap-x-1.5 rounded hover:bg-tertiary pr-1 py-[0.1rem] pl-1.5 transition focus-custom",
"flex items-center h-[1.5rem] gap-x-1.5 rounded hover:bg-tertiary disabled:hover:bg-transparent pr-1 py-[0.1rem] pl-1.5 transition focus-custom disabled:hover:text-charcoal-400 disabled:opacity-50 text-charcoal-400 hover:text-charcoal-200 disabled:hover:cursor-not-allowed hover:cursor-pointer",
root: "h-3 w-6",
thumb: "h-2.5 w-2.5 data-[state=checked]:translate-x-2.5 data-[state=unchecked]:translate-x-0",
text: "text-xs text-charcoal-400 group-hover:text-charcoal-200 hover:cursor-pointer transition",
text: "text-xs transition",
},
};

Expand Down
2 changes: 2 additions & 0 deletions apps/webapp/app/components/runs/TimeFrameFilter.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ export function AbsoluteTimeFrame({
granularity="second"
showNowButton
showClearButton
utc
/>
</div>
<div className="space-y-1">
Expand All @@ -227,6 +228,7 @@ export function AbsoluteTimeFrame({
granularity="second"
showNowButton
showClearButton
utc
/>
</div>
</div>
Expand Down
Loading