Status check for a matrix jobs

[edzis]

I build several subprojects using a job with the matrix strategy. The matrix is dynamic - depends on the branch used. All of those matrix jobs need to succeed for the code to be good to merge. But I fail to find a way how to check the composite status of the branches. Which direction would you recommend?

[BrightRan]

@edzis,

The matrix is dynamic - depends on the branch used.

Do you mean that you set different matrix configurations in the workflow on different branches?

But I fail to find a way how to check the composite status of the branches.

What do you mean about “the composite status of the branches”?
Do you mean that summarize the latest checks statuses of all the branches as a composite status?
If so, maybe you can try to use the repository_dispatch to trigger a workflow to summarize all the checks statuses.

If I missed something, feel free to tell me. And if possible, you also can share your repository with us, so that we can check more detailed configurations related to the workflow.

[ylemkimon]

I’m using a job that needs matrix job to check all matrix jobs are successful. For example:

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        number: [0, 1, 2]
steps:
- run: echo ${{ matrix.number }}

build_success:

runs-on: ubuntu-latest

needs: build

steps:

- run: echo Done!

In the example above, build_success job is only run after all build jobs are successful, so you can check on the result of build_success job.

[edzis]

brightran:

Do you mean that you set different matrix configurations in the workflow on different branches?

Yes.

brightran:

What do you mean about “ the composite status of the branches ”?

PR can be merged only if all of the jobs from the calculated matrix complete successfully. If a single job from the matrix fails, the PR merge should be blocked.

brightran:

use the repository_dispatch to trigger a workflow

I don’t understand where and what would I dispatch and how would that help.

ylemkimon:

I’m using a job that needs matrix job to check all matrix jobs are successful.

This gives false-positives - if one of matrix build jobs fail, the build_success is skipped, which is considered as a pass. See screenshot:

Screen Shot 2020-08-18 at 10.50.141740×792 97.3 KB

brightran:

summarize all the checks statuses

I might generate trivial artifacts from each running matrix job and have a job that checks the existence of those based on matrix, but that sounds over-engineered. Hoping to find something more simple.

[BrightRan]

@edzis ,

The checks of a PR generally include the jobs in the workflows related to the PR, and some checks generated from other GitHub Apps.
The related workflows can be:

1. triggered by the events related to pull request, such as pull_request, pull_request_review, pull_request_review_comment and pull_request_target.

2. triggered by push event when pushing new commits to the head branch (source branch) of the PR. The new commits push also triggers the workflows (if any) run on the activity type ‘synchronize’ of pull_request event if any.

Normally, the workflows run on other branches that not related to the PR will not affect to this PR.

If you want the related checks on the PR must be passed before the PR can be merged, you can set some branch protection rules, and set the related checks as ‘Required’. More details, see “Defining the mergeability of pull requests”.

required-checks995×603 37 KB

[edzis]

@brightran I believe we are not on the same page. I believe it might help for me to share more insights about the setup. Here is a short description and the action setup is shared on https://gist.github.com/edzis/17602cb75eb6346b9096b7b53498a424

Conceptually we have a monorepo with multiple projects (customizers) and a shared codebase used by those projects.
We calculate a matrix based on branch name.
When we work on a specific project, we do the work in a branch named CLIENT-PROJECT/FEATURE. Only that project needs to be tested and deployed on dev - the matrix contains only that project and the dev environment.
When we work on generic features only the tests need to be run but there should be no deployment - the matrix job defines deploy = false and only the onlyTest job is run.
When a PR is merged into master we need to test and deploy all projects to dev - matrix includes all projects and the dev environment.
When a project needs to be deployed to staging or production, we push a staging/CLIENT-PROJECT or production/CLIENT-PROJECT branch - the matrix contains only that project and the specified environment.

Any ideas how I can check if all the jobs defined by the matrix have succeeded?

• I can not specify the parent “Deploy” job as required because that is newer run from the perspective of merge guards.
• I can not specify all the possible jobs resulting from the matrix as most of them are not even run in feature branches.
• I can not specify another job that uses the build job as required because, if one of my matrix jobs fails, that job is a false positive - it is skipped which is considered as a pass by the merge guard.

[hodbn]

Hi @edzis,
IIUIC you want to summarize a matrix job to a single check. This way, you can tick only one checkbox in the protected branch settings of the repository and later matrix entries will only change in the corresponding workflow file.

This is a workaround using a utility job (buildall) after your matrix job (build):

  buildall:
    if: ${{ always() }}
    runs-on: ubuntu-latest
    name: Build (matrix)
    needs: build
    steps:
      - name: Check build matrix status
        if: ${{ needs.build.result != 'success' }}
        run: exit 1

In addition, I think you should have continue-on-error: false on the build job (it’s the default).

[edzis]

Thanks @hodbn, that fixes my issue!

I was not aware of

if: ${{ always() }}

and

if: ${{ needs.build.result != 'success' }}

And I tried continue-on-error: false before with no success, but kept the current setup because when merging into master I don’t want to stop all projects when some fail.

[dudicoco]

@hodbn @edzis this solution does not seem to work.
If one of the matrix jobs failed while another succeeded, needs.build.result value will be success.

[thaysealencar]

Same for me. I keep getting success even though one of the jobs of my matrix failed.

[egmacke]

Is there any update on this?

@dudicoco is correct that if any of the matrix jobs are successful then needs.build.result will be success

I have a matrix build that builds any projects that have changed in our mono-repo. I then want to have a single job that runs after so I can configure the github branch protection to require that single job, which would accumulate the results of the matrix job (i.e. if any of the matrix jobs fail, it would also fail)

I could achieve this by turning off fail-fast but that would be extremely couter productive as projects passing/failing build&test are typically independent of eachother.

[dudicoco]

@egmacke I was able to solve this using the following suggestion: Get status of parallel jobs into a single Webhook payload - #2 by weide-zhou

[thaysealencar]

@dudicoco Thanks so much for sharing the link to this discussion. The solution proposed there with the GH API worked perfectly for me.

[egmacke]

@dudicoco thanks! That looks like a nice solution.

However, since posting, I’ve come up with a different alternative, which is to add an additional step in the matrix job that adds a tag to the pull request if it fails. Then I have a final job that checks if the tag has been added or not.

I’ll have a look and see which approach is better for my use case, but thanks for the link!

For reference for anyone else coming here:

name: CI Checks
on:

pull_request:

branches: [master]

types: [opened, synchronize, reopened]

push:

branches: [master]
jobs:

changes:

runs-on: ubuntu-latest

outputs:

matrix: ${{ steps.filter.outputs.changes }}

steps:

- uses: actions/checkout@v2
  - uses: dorny/paths-filter@v2
    id: filter
    with:
      filters: '.github/project-filters.yml'

  - name: Clean up label
    uses: actions-ecosystem/action-remove-labels@v1
    if: success() && github.event.pull_request
    with:
      labels: build-fail
      fail_on_error: false

ci-checks:

name: CI Checks

runs-on: ubuntu-latest

needs: changes

if: fromJSON(needs.changes.outputs.matrix)[0] != null

strategy:

fail-fast: false

matrix:

project: ${{ fromJSON(needs.changes.outputs.matrix) }}

steps:

- uses: actions/checkout@v2
  - name: Do build/test stuff here
    run: echo "Do build"

  - name: Update label
    uses: actions-ecosystem/action-add-labels@v1
    if: failure()
    with:
      labels: build-fail

ci-build-check:

if: always() && github.event.pull_request

runs-on: ubuntu-latest

name: CI Build Check

needs:

- ci-checks

- changes

steps:

- uses: mheap/github-action-required-labels@v1

with:

mode: exactly

count: 0

labels: build-fail

[richja]

late to the party but looks like result of matrix now works as expected when using: needs.<job_id>.result

so I can have just one single additional job which I can set as required

results:
    if: ${{ always() }}
    runs-on: ubuntu-latest
    name: Final Results
    needs: [build]
    steps:
      - run: |
          result="${{ needs.build.result }}"
          if [[ $result == "success" || $result == "skipped" ]]; then
            exit 0
          else
            exit 1
          fi

[sounisi5011]

@sounisi5011 , is there a way to print no of jobs failed too?

@krishna13052001, we probably can't count the elements in an array (in this case, needs.*.result) using only the built-in functions in the GitHub Workflow. So it might be better to convert the array to a string value with the join( array, optionalSeparator ) or toJSON( value ) functions, then parse it in Bash or Python and count it¹. Filtering of failed jobs is also better done within those scripts.

Footnotes

1. The script specified for the run keyword can be Bash, PowerShell, Python, etc. See the shell parameter. ↩

[StephenHodgson]

I'm not sure I agree this should be the answer.

I think the actual runner itself should give a more generalized status 'success' result, and we shouldn't have to add an additional step just to work around this. Should be fairly trivial to add the logic in the Runner implementation itself. It'll likely be similar to how it decides if a "composite" action is successful.

namespace GitHub.Runner.Worker.Expressions
{
    public sealed class SuccessFunction : Function
    {
        protected sealed override object EvaluateCore(EvaluationContext evaluationContext, out ResultMemory resultMemory)
        {
            resultMemory = null;
            var templateContext = evaluationContext.State as TemplateContext;
            ArgUtil.NotNull(templateContext, nameof(templateContext));
            var executionContext = templateContext.State[nameof(IExecutionContext)] as IExecutionContext;
            ArgUtil.NotNull(executionContext, nameof(executionContext));

            // Decide based on 'action_status' for composite MAIN steps and 'job.status' for pre, post and job-level steps
            var isCompositeMainStep = executionContext.IsEmbedded && executionContext.Stage == ActionRunStage.Main;
            if (isCompositeMainStep)
            {
                ActionResult actionStatus = EnumUtil.TryParse<ActionResult>(executionContext.GetGitHubContext("action_status")) ?? ActionResult.Success;
                return actionStatus == ActionResult.Success;
            }
            else
            {
                ActionResult jobStatus = executionContext.JobContext.Status ?? ActionResult.Success;
                return jobStatus == ActionResult.Success;
            }
        }
    }
}

[StephenHodgson]

Digging into this a bit more. It seems that the results are only merged into the larger job context if any single step fails:

                // Update the job result
                if (step.ExecutionContext.Result == TaskResult.Failed)
                {
                    Trace.Info($"Update job result with current step result '{step.ExecutionContext.Result}'.");
                    jobContext.Result = TaskResultUtil.MergeTaskResults(jobContext.Result, step.ExecutionContext.Result.Value);
                    jobContext.JobContext.Status = jobContext.Result?.ToActionResult();
                }
                else
                {
                    Trace.Info($"No need for updating job result with current step result '{step.ExecutionContext.Result}'.");
                }

                Trace.Info($"Current state: job state = '{jobContext.Result}'");

[StephenHodgson]

It could also be that they're only listening to ActionsStepTelemetry being sent and never sends the same kind of telemetry for successful Job completion. In either case it'd be nice to have that as well.

[ju-manns]

This does not seem to work either. If multiple jobs of the matrix fail and a user reruns a single job in that matrix and that single job succeeds, the results job will also be successful.

[ju-manns]

@richja's solution has the flaw that manually rerunning a single matrix job (in the case of multiple failures) will make the results job pass as long as that single job passes.
I see two ways to fix this:

1. Add an aggregation job before the results job. This job only runs if the matrix job ran. Then check if the matrix job was skipped or if the aggregate job was successful.

aggregate:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - run: exit 0

results:
    runs-on: ubuntu-latest
    needs: [build, aggregate]
    if: ${{ !cancelled() }}
    steps:
      - if: ${{ needs.build.result != 'skipped' && needs.aggregate.result != 'success' }}
        run: exit 1
      - run: exit 0

2. Remove the if: ${{ always() }} condition from the results job:

results:
    runs-on: ubuntu-latest
    name: Final Results
    needs: [build]
    steps:
      - run: exit 0

The second option has the flaw that the status check is only fufilled if the matrix job actually runs. A dynamically created matrix job might have an empty matrix though and adding a dummy job that always runs to each matrix job is cumbersome.

All of this is pretty hacky. Github actions needs an internal solution for this.