Autobuild fails intermittently on Ubuntu 20.04

[jasonleenaylor]

We have noticed that in our CodeQL github action the autobuild fails a little less than 10% of the time.

When it succeeds the dotnet build --no-incremental /p:UseSharedCompilation=false completes without error, but when it fails the logs show a successful build but there is a non 0 exit code.
The fallback to MSBuild doesn't work for our project so the autobuild eventually fails.

So far the job has always succeeded when we re-run it.

Here is one example of the failure in our project:
https://github.com/sillsdev/TheCombine/runs/3204859583#step:4:157

[rdmarsh2]

Hi, I've passed this on to our C# team. In the meantime, would you mind adding this snippet to your CodeQL workflow after the finalize step to upload some extra logs as Actions artifacts for failed jobs?

- uses: actions/upload-artifact@v2
  if: ${{ failure() }}
  with:
    name: tracer-logs
    path: ${{ runner.temp }}/*.log

[jasonleenaylor]

@rdmarsh2 We now have a failure with the trace logs to view here:
https://github.com/sillsdev/TheCombine/pull/1300/checks?check_run_id=3295392201

Hmm, re-running it seems to have removed the Artifact, but I have a copy if anyone wants it. It doesn't actually appear to have anything useful to my eyes.

[jasonleenaylor]

@rdmarsh2 We had another failure today: https://github.com/sillsdev/TheCombine/pull/1307/checks?check_run_id=3325019354 Would anyone on the C# team have any more suggestions on how to diagnose this?

[criemen]

@jasonleenaylor I was able to reproduce this by building your project in a loop, after ~40min I got the crash.
I will discuss this with the team and get back to you how we will address this problem.

[jasonleenaylor]

@criemen We just had this pop up again today. I appreciate that you all made the effort, any updates?

[johnthagen]

The latest example from today: https://github.com/sillsdev/TheCombine/runs/3505515280

[criemen]

@jasonleenaylor I am sorry for not getting back to you, I simply forgot to update the issue after discussing this internally.

We are currently working on a rewrite of some parts of our build tracing infrastructure that will allow us to completly get rid of the piece of software that is crashing right now.
Unfortunately, that release is still 1-2 months out, and there's no workaround for this except to re-run the failing workflow.

[jasonleenaylor]

@criemen I hope the rewrite is going smoothly. We hit it again today.

[jasonleenaylor]

@criemen This popped up again for us so I came to check in. Am I right in assuming that the discussion in PR #770 is of interest to us? 🤞

[criemen]

I am sorry to hear!
We are still developing a solution to this problem. It is underway, it's just taking some time.
Unfortunately, the new-style tracing refered to in #770 is not enough (but indeed a part) of that solution.

[criemen]

I just wanted to pop in and say we've not forgotten you!
Addressing this issue got postponed for making CodeQL compatible with Windows 11, but now we're back on track.
Hopefully, we'll be able to ship a fix for this issue around June/July.

[jasonleenaylor]

@criemen We've had a bunch of failures recently. How is it going?