Parallel Steps #14484
Replies: 29 comments · 48 replies
-
|
I'd like that too. There are scenarios where it doesn't make sense juggling artifacts between jobs. An idea how it could be used: steps:
- uses: actions/checkout@v3
- run: echo "Sequential step"
- parallel:
branch_id_1:
steps:
- run: echo "branch 1 sequential step 1"
- run: echo "branch 1 sequential step 2"
# Executes steps in parallel to branch1
branch_id_2:
steps:
- run: echo "branch 2 sequential step 1"
- run: echo "branch 2 sequential step 2"
- run: echo "More sequential steps after the whole parallel part is done." |
Beta Was this translation helpful? Give feedback.
-
|
This is a good approach as it mirrors Jenkins well-known solution for parallelizing stages. |
Beta Was this translation helpful? Give feedback.
-
|
Response #1 |
Beta Was this translation helpful? Give feedback.
-
|
just bumping https://github.com/qoomon/actions--parallel-steps https://github.com/orgs/community/discussions/14484#discussioncomment-12893306 |
Beta Was this translation helpful? Give feedback.
-
|
Alternative solution: |
Beta Was this translation helpful? Give feedback.
-
Alternative solution: |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I'm missed how putting steps in a composite action would allow you to parallelize steps such as multiple logins to different registries? Could you please elaborate further with regards to how this solves step parallelization? |
Beta Was this translation helpful? Give feedback.
-
|
Parallel steps or sub jobs would save us 50% of time waiting for the CI. |
Beta Was this translation helpful? Give feedback.
-
|
GH would lose a lot of money if they let people halve the time they charge for. |
Beta Was this translation helpful? Give feedback.
-
|
Logging into docker in parallel is probably going to result in things exploding. Each docker login command tries to mutate Today, you could probably do something like: (To make these work, you'd want to have them rely on env vars and set the env vars using your workflow based on secrets)
HOME=$HOME/.docker-docker
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-gcr
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-ghcr
mkdir -p $HOME
docker login ...
for task in ./log-into-*.sh; do
$task &
done
wait
jq '. * input' ~/.docker*/config.js > my.config.js && mv my.config.js ~/.docker/config.js
jq '. * input' ~/.docker*/plaintext-credentials.config.json > my.plaintext.js && mv my.plaintext.js ~/.docker/plaintext-credentials.config.json |
Beta Was this translation helpful? Give feedback.
-
|
If the |
Beta Was this translation helpful? Give feedback.
-
|
Talking to the docker folks is an exercise for the brave. I wouldn't recommend it to my worst enemy. |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref I think instead of for loop with
About docker I totally agree. Talking to docker devs is PITA and nothing that improve dev work is actually done e.g. multiple dockerignore files or mounting volumes as non root to container and so on. |
Beta Was this translation helpful? Give feedback.
-
|
@piotrekkr: thanks on both points. |
Beta Was this translation helpful? Give feedback.
-
|
can we have the parallel steps please |
Beta Was this translation helpful? Give feedback.
-
|
YES, this is the most useful option and it's obvious. Downvote must come from some GitHub dev who does not like to have some work or what? (nah, downvote was about something else, sorry, it's valid) |
Beta Was this translation helpful? Give feedback.
-
Nope, it's coming from me, because this answer doesn't bring any value. Basically a spam. |
Beta Was this translation helpful? Give feedback.
-
|
Because here no-one can request for anything because GitHub action devs don't see it? Okay, I'm not aware where requests can be made. It seems to be anyways useless because there are some many open requests at various pages and I also saw GitHub action devs saying, they will probably not do it |
Beta Was this translation helpful? Give feedback.
-
|
The main post is the by
Unfortunately yes :(
Link please? |
Beta Was this translation helpful? Give feedback.
-
|
It's from memory, so cannot tell, sorry. I think it was either about better caching or about their unwillingness to make "include yml files for steps" or some other of the obvious shortcomings. But then again, not sure if Github even claims to be a CI/CD solution or just code repo with benefits, so maybe we should not be so hard |
Beta Was this translation helpful? Give feedback.
-
|
The BitBucket has parallel step options. Why not add this option to GitHub actions? |
Beta Was this translation helpful? Give feedback.
-
|
This is the biggest pain point we've run into coming from Jenkins. You can approximate this feature with |
Beta Was this translation helpful? Give feedback.
-
|
How come it's error prone? |
Beta Was this translation helpful? Give feedback.
-
|
Every project has different needs about what artifacts they need available, how they're shaped, fs permissions, size costs, etc... |
Beta Was this translation helpful? Give feedback.
-
|
Need parallel steps support for our scenario: deploy multi clusters concurrently within the same VPN environment. So it is convinient to achive in steps rather than actions or jobs |
Beta Was this translation helpful? Give feedback.
-
|
Using a yarn monorepo I'd like to:
yarn install is slow, and I currently have to repeat it for every workflow like |
Beta Was this translation helpful? Give feedback.
-
|
We've been iterating over an action that does some of the work to make this better: https://github.com/GarnerCorp/build-actions/blob/main/yarn/action.yml You can use actions/upload-artifact and actions/download-artifact to share a build product (e.g. the output tarball from the above action) between jobs (especially a matrix of jobs to run tests). |
Beta Was this translation helpful? Give feedback.
-
|
Are you using yarn workspaces? If you are, you should be able to run |
Beta Was this translation helpful? Give feedback.
-
|
I was looking for a solution for exactly the same problem @raphaelbadia. I did get caching to work but I'm using PNPM, which I would highly recommend as an easy time saver compared to Yarn. It uses symlinks to a system level package cache, instead of installing dependencies in all your monorepo projects. |
Beta Was this translation helpful? Give feedback.
-
|
Check out this approach: Making pnpm run jobs in parallel instead of finding a way to run GitHub steps in parallel: https://stackoverflow.com/questions/68427503/how-can-i-run-pnpm-workspace-projects-as-parallel-jobs-on-github-actions |
Beta Was this translation helpful? Give feedback.
-
|
@michielvanderros-tomtom speaking of PNPM :) Yarn supports pnpm linker out of the box, in addition to pnp and regular node_modules; yet it's developed by a team vs. single person behind pnpm, and has great design with plugin system. |
Beta Was this translation helpful? Give feedback.
-
|
If its easier, GitHub could also allow sharing jobs workdirs |
Beta Was this translation helpful? Give feedback.
-
|
You can use |
Beta Was this translation helpful? Give feedback.
-
|
I'm aware of that. |
Beta Was this translation helpful? Give feedback.
-
|
Have a look here: https://circleci.com/docs/caching/ |
Beta Was this translation helpful? Give feedback.
-
It's super slow. Impractical for anything larger. |
Beta Was this translation helpful? Give feedback.
-
|
I'm in the minority, but I'd actually like this way better too— the handling of logs is more natural because each job has its own, and you already have the dag visualization of how the jobs fit together. The obvious syntax would be instead of Main question mark would be what happens to those post-action steps, if you've got a bunch of torn down auth state, that's mildly annoying, though re-authing is not nearly as bad as starting from scratch. This doesn't cover the "multiple preparatory steps in parallel" ask that some others have been making, but I'm honestly not sure if that should be the responsibility of GitHub Actions anyway. Yes, you can do that kind of thing in your Jenkinsfile, but really it belongs in a proper build tool. Let |
Beta Was this translation helpful? Give feedback.
-
|
I just want to start a resource in a background, which needs some time to start, eg a docker container or a simulator. This resource should start during the job start. My test step should wait until the test resource is available, or fail after an (optional) timeout. - jobs:
- build
- resources:
- test-db
run: docker start someSlowImage
- steps:
- run: assemble
- run: test
resources:
- test-db
timeout: 5min |
Beta Was this translation helpful? Give feedback.
-
|
I think you can do this using service containers: https://docs.github.com/en/actions/using-containerized-services/about-service-containers |
Beta Was this translation helpful? Give feedback.
-
|
GitLab CI does a similar thing: https://docs.gitlab.com/ci/services/ |
Beta Was this translation helpful? Give feedback.
-
Just add
|
Beta Was this translation helpful? Give feedback.
-
|
They might not, but each Google Cloudbuild step runs a separate disposable container and you can't persist state from one to another. Thus anyone making a comparison to them is making a very uniformed comparison. I'm well aware of both. We're in the process of migrating some builds from Cloudbuild to GitHub Workflows and I've been using GitHub Workflows/Actions for years. |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref you can pass state between steps in Cloud Build as easily as in GitHub Actions steps (docs). For example, the following list of steps — steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
# Install dependencies
- run: yarn install
# Build the API and web front-end (app)
- id: build-api
run: yarn workspace api build
- id: build-app
run: yarn workspace app build
# Deploy
- id: deploy-api
run: yarn workspace api deploy
waitFor: ["build-api", "build-app"] # <- This "waitFor" field demonstrates
- id: deploy-app # the elegance of parallel steps feature
run: yarn workspace app deploy # in Google Cloud Build. If implemented
waitFor: ["build-api", "build-app"] # in GitHub Actions it would reduce the pain
# configuring parallel steps 10-fold 💰 🙌 👍 🎉 👏 💪^^ Deployment of packages |
Beta Was this translation helpful? Give feedback.
-
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
# Install dependencies
- run: yarn install
# Build the API and web front-end (app)
- id: build-api-app
run: |
yarn workspace api build &
yarn workspace app build
fg %1 || true
# Deploy
- id: deploy-api-app
run: |
yarn workspace api deploy &
yarn workspace app deploy
fg %1 || true |
Beta Was this translation helpful? Give feedback.
-
|
@jsoref In your example above there are no separate steps for building individual packages/projects, with separate success/failure statuses and separate logs, that's the point of this issue. I can't stress enough how painful it is (at least for me) to have this feature (parallel steps) missing in GitHub Actions. 😫 😖 😣 😡 💔 😭 😩 💢 🤬 💥 💢 😤 |
Beta Was this translation helpful? Give feedback.
-
|
You can make that if you want, just create logs for each thing in the main step and then pass the various job outputs to individual steps for reporting. It really isn't terribly hard |
Beta Was this translation helpful? Give feedback.
-
|
As a workaround I was able to use this ` name: Distributed Tasks This way I was able to run our tests parallel |
Beta Was this translation helpful? Give feedback.
-
|
Are you able to see the logs separately for different parallel commands in GitHub Actions? |
Beta Was this translation helpful? Give feedback.
-
|
@koistya : no, can not see logs separately , but if one of the cmd fails the can see logs |
Beta Was this translation helpful? Give feedback.
-
|
That’s an interesting workaround, @malindajudo. Running parallel steps in GitHub Actions can definitely simplify workflows, especially when dealing with tasks like multiple deployments or integrating third-party APIs. For example, in a web app development company, managing Docker containers and parallelizing tasks can help eliminate the complexity of sequential executions. This capability would allow teams to scale their processes faster, reduce bottlenecks, and enhance overall efficiency in their CI/CD pipelines. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Hi. Are there any updates on this topic from a Github team member ? |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Would be great to have some updates here. Parallel steps would be great, plenty of instances where for example I'd like to know whether multiple steps fail (eg: a lint check as well as unit tests). While jobs can run in parallel, each needs to be setup again. Would be great for a pipeline to be able to run a setup step that for example pulls dependencies and then splits off in parallel steps each starting from the state left by the setup. |
Beta Was this translation helpful? Give feedback.
-
|
Would be awesome if we can, for example, compile our js/css files at the same time we are running composer (for PHP projects). |
Beta Was this translation helpful? Give feedback.
-
|
Just wanted to add, that parallel jobs means waiting for multiple available workers, which can add a lot to the total build time, and in some cases you don't need the extra hardware. Not sure what's the problem with implementing this. It just adds an option that aparently a lot of people find useful. |
Beta Was this translation helpful? Give feedback.
-
Bitbucket can do it, just saying... |
Beta Was this translation helpful? Give feedback.
-
|
the current way of running steps in parallel (i.e using bash |
Beta Was this translation helpful? Give feedback.
-
|
Don't know much about Actions as I'm new to this (trying to find alternatives to Drone CI), but a simple feature like this took so long time to implement, maybe the design of Actions has some fundamental flaws on create DAG of steps.
Drone steps are by default parallel, unless you add a 'depends_on' list to steps, which would transform them to a DAG. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I was considering Drone CI to use parallel steps, any reasons why you're trying to replace it? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Would love this feature as well. We already have quite long running pipelines and saving some time would be great. We build our .NET backend then pack it as docker image. While creating the image takes significant time we also need to upload our symbols to Sentry for better error analysis. While docker images are being created we could parallelize the upload to Sentry. A new job is no option as we lose the symbol files we require. Right now we run a separate job which fetches git, compiles, etc. all just to upload already built symbols. Please implement this! :) |
Beta Was this translation helpful? Give feedback.
-
|
I have 5 small steps (10-30sec), each runs a docker container with different arguments. Ideally I would have first step to build docker image and then run those five steps in parallel. Doing this with workflows seems cumbersome and would possibly prolong the total run time with artifact upload and artifact downloads added to share the image between jobs. |
Beta Was this translation helpful? Give feedback.
-
|
if you can fit more than one docker container in a single github runner, then it definitely makes more sense to use a single step to create the image and then run the steps in parallel. We use |
Beta Was this translation helpful? Give feedback.
-
correct me if I'm wrong, but in a case with docker compose that would be a single step and you have to dig into the logs to figure out which of the containers failed? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. We did it. It isn't difficult. You could probably use |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Hey y'all! We are still gathering all the high engagement feedback items and triaging them for our work this year. Sorry I haven't commented on this one already. Thank you for all engaging and staying engaged for such a long period of time - please keep voting up on the top request and sharing your feedback with us |
Beta Was this translation helpful? Give feedback.
-
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
Beta Was this translation helpful? Give feedback.
-
|
I just released https://github.com/qoomon/actions--parallel-steps This action supports basically all types of steps. |
Beta Was this translation helpful? Give feedback.
-
|
the latest release now supports pre and post steps and command files (except step summary) |
Beta Was this translation helpful? Give feedback.
-
|
With the latest release of act https://github.com/qoomon/actions--parallel-steps is now fully functional. Looking forward for some feedback |
Beta Was this translation helpful? Give feedback.
-
|
@HariSekhon does this work for your initial post/request? |
Beta Was this translation helpful? Give feedback.
-
|
Just adding my voice so maybe by the time my grandkids are coding, GitHub Actions will support parallel steps. |
Beta Was this translation helpful? Give feedback.
-
|
Implementing parallel steps in GitHub Actions would significantly enhance CI/CD workflows, especially for tasks like deploying to multiple environments or managing dependencies across various services. For instance, in a mobile app development company, managing Docker containers and parallelizing tasks can help eliminate the complexity of sequential executions. This approach not only speeds up the development cycle but also ensures more efficient use of resources. Incorporating a feature that allows for parallel execution of steps within a single job, while maintaining access to the same environment and cache, would be a valuable addition. It would streamline processes and reduce the overhead associated with managing multiple jobs. Looking forward to seeing this feature implemented, as it would greatly improve the flexibility and efficiency of workflows in GitHub Actions. |
Beta Was this translation helpful? Give feedback.
-
|
try my action https://github.com/qoomon/actions--parallel-steps |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
Can you please raise a request to engineering for parallel steps support?
Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects.
Uploading/downloading artifacts from adjacent jobs is not suitable for this use case.
Thanks
Hari
Beta Was this translation helpful? Give feedback.
All reactions