Support upload of large databases (> 2GB)

[aibaars]

Currently the codeql-action crashes when trying to upload a CodeQL database bundle that is larger than 2GB. This is because the codeql-action tries to load the entire zip file into a Buffer which is limited to 2GB in Javascript.

We should probably rewrite:

codeql-action/src/database-upload.ts

Lines 43 to 45 in 4fddc51

	const payload = fs.readFileSync(
	await bundleDb(config, language, codeql, language)
	);

and supply the payload to the POST request as a streaming input.

Creating zip file at C:\a\_temp\codeql_databases\csharp.zip.
Error: File size (8048141762) is greater than 2 GB
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (8048141762) is greater than 2 GB
    at new NodeError (node:internal/errors:371:5)
    at tryCreateBuffer (node:fs:419:13)
    at Object.readFileSync (node:fs:464:14)
--->at uploadDatabases (C:\a\_actions\github\codeql-action\v2\lib\database-upload.js:51:28)<---
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:184:9)
    at async runWrapper (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:235:9) {
  code: 'ERR_FS_FILE_TOO_LARGE'

[aeisenberg]

What is the max db size that turboscan can handle?

[rneatherway]

We believe that the limit is 5GB due to the need to use multipart uploads above that size. However, we haven't tested around those sizes yet.

[robertbrignull]

This particular issue should be fixed now. Though note that uploads of multi-gigabyte databases is still not fully supported, so it will currently fail for other reasons. We'll be working on improving this over time so larger databases can be uploaded.