Azure Pipelines: `macOS CI Tests` and `Ubuntu CI Tests (coverage)` jobs are always skipped

[sobolevn]

I cannot find any CI runs where these two are completed / failed. They are always skipped.
Looks like Azure only stores build result for ~1 month or so, so I cannot trace it back further.

Started: Today at 09:12
Duration: 2h 3m 40s
Evaluating: False
Result: False

Examples:

• https://dev.azure.com/Python/cpython/_build/results?buildId=137735&view=logs&j=609bc2f2-68da-5434-7493-ff39ba61961e
• https://dev.azure.com/Python/cpython/_build/results?buildId=135386&view=logs&j=4fbfd83d-b707-5326-9192-5affa2b62e03
• https://dev.azure.com/Python/cpython/_build/results?buildId=137732&view=logs&j=4fbfd83d-b707-5326-9192-5affa2b62e03

Why does it happen?

Linked PRs

• gh-109395: Remove skipped macOS builds from Azure Pipelines #109400
• gh-109395: Remove skipped coverage job from Azure Pipelines #109412
• [3.12] gh-109395: Remove skipped coverage job from Azure Pipelines (GH-109412) #109433
• [3.11] gh-109395: Remove skipped coverage job from Azure Pipelines (GH-109412) #109434
• [3.12] gh-109395: Remove skipped macOS builds from Azure Pipelines (GH-109400) #109441
• [3.11] gh-109395: Remove skipped macOS builds from Azure Pipelines (GH-109400) #109442

[hugovk]

macOS CI Tests

The macOS tests were disabled 3.5 years ago, when GitHub Actions CI was newer: PR #18818.

Ubuntu CI Tests (coverage)

This is skipped because the condition:

    and(
        and(
            succeeded(),
            eq(variables['coverage'], 'true')
        ),
        eq(dependencies.Prebuild.outputs['tests.run'], 'true')
    )

evaluates to false:

    and(
        and(
            True, 
            eq('false', 'true')
        ),
        eq(dependencies['Prebuild']['outputs']['tests.run'], 'true')
    )

Because variables['coverage'] is 'false'.

There is a coverage: true for the job's only step, but it looks like the above condition is evaluating to false before the step can even run. I've not checked if this is intentional.

---

Are these Azure Pipelines runs doing anything that is not covered by the GitHub Actions CI?

If not, a better course of action would be to remove them from Azure Pipelines. GitHub Actions is easier to manage, especially when it comes to restarting and maintenance. (Compare: PR #105823 that removed docs testing from AP.)

(The same applies to the other Ubuntu and Windows jobs that are running, but there's at least some unique things run in patchcheck that hasn't been migrated over, so we can deal with these later.)

---

In the issue that led to disabling the macOS jobs, @zooba outlined the required steps (#84018 (comment)):

Deleting the files is not the right first step.

First, it needs to be changed to a non-required check.

Then, I can use the web UI to disable it starting.

Then, we can remove some of the files in the directory. Others are used for the official release, and have to stay.

@zooba, for macOS:

• change to a non-required check (bpo-39837: Disable macOS tests on Azure Pipelines #18818)
• disable in web UI - does this need doing?
• remove some files: can we delete .azure-pipelines/macos-steps.yml (and the matching jobs in ci.yml and pr.yml)?

For Ubuntu coverage:

• First off, do we want to fix coverage?
• If so, should we add something to GitHub Actions first?
• In any case, can we remove this skipped job now?

[sobolevn]

@hugovk thanks a lot for the context. I've sent #109400 to see what will happen if macos-steps.yml is deleted.

Looks like it should be fine, but we will see :)

[hugovk]

Ubuntu CI Tests (coverage)

A quick experiment that hardcodes coverage: true to check if the run completes shows it core dumped after 59 minutes:

• 04f7fdb

• https://dev.azure.com/hugovk/CPython/_build/results?buildId=4009&view=logs&j=dab3c75d-0b55-5e4b-43c6-bb67925e6034&t=c144e935-82e9-5e97-ddc9-7022ba754539

Compared to 15 minutes for the regular Ubuntu CI Tests build which passes.

Here's a PR to remove the skipped coverage job: #109412

[zooba]

Yeah, coverage just never worked, unfortunately. It was a nice idea. It's possible that the segfault at 59 minutes is just the 1 hour timeout sending a Ctrl+C at a bad time, but the tests are so slow I don't think we'd want it enabled anyway. Maybe as a scheduled (daily) job.

Removing the files is fine. They don't do any harm by staying there, and might be helpful to someone trying to run tests on their fork, but if people are going to keep suggesting we remove them then I guess we may as well make them happy.

[hugovk]

At work, the coverage is also slow on the CI. But it's faster locally, so we sometimes use it for local development. And you might just run a subset of tests for the bit of code you're working on, and check you're properly covering your new code.

So something for the future may be to set tooling up for local development, if people would find that useful.