Other considerations that will need some more in-depth exploration:

• Multi-language support (e.g. python). Sort of like how Vercel supports Python runtime for serverless. Could be very useful for ML tasks
• Integration support. It might be nice/required for these background tasks to support integrations, similar to how jobs do. This would allow background tasks to have access to hosted credentials (e.g. OAuth access tokens) and eventually allow them to be run as a user (via Trigger.dev Connect)

Background Task Function Alpha Proposal

To review, our goals with the first release of this feature are:

• Get something shipped quickly to evaluate use-cases and iterate on fixes & improvements
• Be usable by self-hosters without having to run a large infrastructure project
• Provide a really good DX that fits in with the existing Trigger.dev

Our initial idea was to build a multi-tenant offering that would be powered by Fly.io, but Trigger.dev would manage building and deploying docker images for Background Task Functions*, but after some initial development we've run into a few realities:

• Building a multi-tenant offering on top of Fly.io is still a lot of work
• Self-hosters don't really need multi-tenant support, now and into the future

Alpha proposal

We're proposing a new Alpha version of this feature:

• A npx @trigger.dev/cli build command that would build and publish a hostable docker image for each background function called a Background Function Worker
• This docker image will then be run by users on whatever platform they wish (more on platforms below)
• Trigger.dev would not orchestrate the running of this image, it would be up to the user
• Docker image builds would happen in Depot.dev and would be handled by Trigger.dev
  • This allows reliable docker images to be produced without platform issues (i.e. building on arm64 macs)
  • Self-hosters using this feature would need to signup to Depot.dev and add API keys to the env vars
• On the Trigger.dev cloud, function images would be hosted at a Docker Registry at registry.trigger.dev
  • For deploying to platforms that don't support registry.trigger.dev, images can be pulled and re-pushed to Docker Hub, etc.
• For self-hosters, they could put in their own Docker Registry url and credentials using environment variables

Platform support

• Render.com - supports deploying prebuilt images from Docker Hub and GitHub/Gitlab Container Registries (https://render.com/docs/deploy-an-image)
• Fly.io - supports deploying from Docker hub and registry.fly.io (https://community.fly.io/t/deploy-locally-built-image/2951)
• Railway.app - Does NOT support deploying pre-built images
• AWS ECS - supports deploying docker images from pretty much any repository

DX

This is how it would all work in practice:

First, you would define a background task in your code as a default export in a file with .background.ts extension:

// src/functions/function-1.background.ts
import { client } from "@/trigger";
import { z } from "zod";

export default client.defineBackgroundFunction({
  id: "function-1",
  name: "Function 1",
  version: "1.0.2",
  schema: z.object({
    userName: z.string(),
  }),
  run: async (payload) => {
    // This code will run in the background, as it will be bundled and shipped to a trigger.dev background worker
    await new Promise((resolve) => setTimeout(resolve, 100000));

    return {
      username: payload.userName,
      foo: "bar",
      message: `Task Response for user ${payload.userName}`,
    };
  },
});

Then in a Job.run function, you would invoke the function like so:

// src/jobs/examples.ts
import { eventTrigger } from "@trigger.dev/sdk";
import { client } from "@/trigger";
import function1 from "@/functions/function-1.background";

client.defineJob({
  id: "example-job",
  name: "Background Function Usage",
  version: "0.0.1",
  trigger: eventTrigger({
    name: "example.event",
  }),
  run: async (payload, io, ctx) => {
    const output = await function1.invoke("task-1", {
      userName: "ericallam",
    });

    return { output };
  },
});

When the example-job Job runs, a task will be created and shown in the dashboard:

When deploying to production, you would use the build command to build and push Background Function Worker docker images for each background function:

$ npx @trigger.dev/cli functions build
Built function-1@1.0.2: registry.trigger.dev/clmgatoar0003dyc44r2kvb0h-function-1:v1.0.2@sha256:91f60ea55704db7162a14ac18387dd31e77963b752a40dd2dfdd058541c25ce4

Environment variables

Because it's not secure to store sensitive data in docker images, users will need to supply any necessary environment variables when they deploy these Background Function Worker images. The only Trigger.dev related environment variable they'll need to include is TRIGGER_API_KEY.

Development

When running locally during development, there is a spectrum of the experience we could provide:

1. Functions would get immediately invoked and run, instead of queued to be run by a Background Function Worker.
2. The @trigger.dev/cli dev command could automatically build Background Function Workers and run them on the users machine via the Docker Daemon
3. The @trigger.dev/cli dev command could automatically build a non-docker version of a Background Function Worker and run them on the users machine via node.js child processes and a custom runtime just for development (that would closely mimic the production one embedded in Background Function Worker images

• We're going to rename this feature from Background Tasks to Background Functions, so we don't overuse the "task" concept which already exists in Job runs

Background Function Library proposal

Before we expose the ability for end-users to develop their own custom Background Functions, we're going to implement a common library of internal background functions that can be used by users without any additional work.

We currently have ad-hoc support for Background Fetch which does a Fetch outside of developers apps and instead performs the fetch on the Trigger.dev platform. Once the fetch response completes (either successfully or after retrying) then the "task" that initiated the background fetch is completed and the run is resumed.

The Background Function Library will move this ad-hoc "Background Fetch" into just 1 of many different background functions that can be referenced and called within Job runs and within integrations. In addition to Background Fetch, we could offer the following functions:

• Bulk Image processing
• Video/Audio encoding or transcribing
• Puppeteer tasks (i.e. taking screenshots)
• ML tasks

The Background Function Library would be implemented in a way that would prepare for opening up the ability for users to define their own background functions (see my previous comment for more on that).

This would allow us to work out bugs and the experience of background functions before we open it up to end users. It would also give us the ability to satisfy more users who want to perform certain long-running tasks quicker.

Considerations for self-hosters

This will add an additional docker image to run for self-hosters called trigger.dev-core-functions, that would include the runtime for all the Background Function library. We may also produce a separate docker image for each core function, allowing independent scaling/resources depending on the function (e.g. the ffmpeg function will need difference resource requirements and dependencies than the background fetch function)

We considered not creating another docker image for this, and instead adding the runtime for the Background Function library into the main trigger.dev docker image, but didn't think there would be much benefit to this as it would overload the image and it's tricky to run multi-process docker images and could lead to issues. But this is still something we haven't 100% decided on.

DX

Currently, using background fetch looks like this:

import { eventTrigger } from "@trigger.dev/sdk";
import { client } from "@/trigger";

client.defineJob({
  id: "function-usage-1",
  name: "Background Function Usage",
  version: "0.0.1",
  trigger: eventTrigger({
    name: "example.event",
  }),
  run: async (payload, io, ctx) => {
    const output = await io.backgroundFetch("fetch-1", "https://example.api", {
      method: "POST",
      body: JSON.stringify(payload),
    });

    return { output };
  },
});

In the new version background fetch would be imported and invoked from the @trigger.dev/functions package:

import { eventTrigger } from "@trigger.dev/sdk";
import { client } from "@/trigger";
import { fetchFunction } from "@trigger.dev/functions";

client.defineJob({
  id: "function-usage-1",
  name: "Background Function Usage",
  version: "0.0.1",
  trigger: eventTrigger({
    name: "example.event",
  }),
  run: async (payload, io, ctx) => {
    const output = await fetchFunction.invoke("fetch-1", "https://example.api", {
      method: "POST",
      body: JSON.stringify(payload),
    });

    return { output };
  },
});

Code Structure

The following packages/apps would be added to the monorepo:

• @trigger.dev/functions: the library of common background functions
• @trigger.dev/functions-worker: orchestrates the running of background function tasks and reporting status/logs/metrics to the Trigger.dev platform
• apps/core-functions: a new app/docker image that would use @trigger.dev/functions and @trigger.dev/functions-worker to run the background function tasks

This feature is what pipedream special. Anyone know any similar product in zapier's space had this feature?

Their Node, Python, Bash & Go runtimes, is extremely feasible and amazing.

I'm going to close this discussion as it's superseded by what we're going to be calling v3.

Exciting proposal! Have you considered leveraging Crawlbase for web scraping tasks within your Background Functions? It could seamlessly integrate with your existing setup, offering reliable and scalable web scraping capabilities. Plus, it aligns with your aspiration of an easy-to-use solution while enhancing task diversity. Just a thought!