Project References Core Support#22420
Project References Core Support#22420RyanCavanaugh wants to merge 35 commits intomicrosoft:masterfrom
Conversation
|
As discussed offline, we need to handle scenarios with two |
97a93b1 to
f83ad4c
Compare
src/compiler/commandLineParser.ts
Outdated
| category: Diagnostics.Module_Resolution_Options, | ||
| description: Diagnostics.Type_declaration_files_to_be_included_in_compilation | ||
| }, | ||
| { |
There was a problem hiding this comment.
i think this should be one level higher. it is similar to includes, excludes and files more than a compiler option.
There was a problem hiding this comment.
Additionally, having a references list implies composite, right? Can we combine the two, eg composite: [ /**/ ] for stuff with dependencies but composite: true or composite: [] for leaf nodes?
Edit from @RyanCavanaugh as I can't reply to this (???) - recycling bits:
References does not imply composite. Leaf-node projects may choose to have references without being composite (for example, it makes no sense to be forced to emit tsc.d.ts, but tsc's tsconfig reasonably has references to checker, etc.).
There was a problem hiding this comment.
one other reason why this should not be here is do not think we want tsc --references a\tsconfig.json to work... or is that a scenario we should support?
There was a problem hiding this comment.
👍 for moving this up a level.
References does not imply composite. Leaf-node projects may choose to have references without being composite (for example, it makes no sense to be forced to emit tsc.d.ts, but tsc's tsconfig reasonably has references to checker, etc.).
Definitely no references on the commandline.
src/compiler/commandLineParser.ts
Outdated
| type: "string" | ||
| } | ||
| }, | ||
| { |
There was a problem hiding this comment.
so we have this in both places?
There was a problem hiding this comment.
Removed option from compilerOptions
| context += resetEscapeSequence; | ||
| } | ||
|
|
||
| output += formatColorAndReset(relativeFileName, ForegroundColorEscapeSequences.Cyan); |
There was a problem hiding this comment.
so why did we change these?
There was a problem hiding this comment.
I'm guessing merge error or some kind?
src/compiler/program.ts
Outdated
| Debug.assert(!!missingFilePaths); | ||
|
|
||
| // List of collected files is complete; validate exhautiveness if this is a project with a file list | ||
| if (options.composite && rootNames.length < files.length) { |
There was a problem hiding this comment.
we have done most of these project validation type checks in verifyCompilerOptions
src/compiler/program.ts
Outdated
| // Project compilations never infer their root from the input source paths | ||
| commonSourceDirectory = getNormalizedAbsolutePath(options.rootDir || getDirectoryPath(normalizeSlashes(options.configFilePath)), currentDirectory); | ||
| } | ||
| else if (options.rootDir && checkSourceFilesBelongToPath(emittedFiles, options.rootDir)) { |
There was a problem hiding this comment.
so what happenes if we set it if rootDir is set? we will report the error later on any ways..
the reason i ask, is it will make more sense to write it as:
if (options.rootDir) {
commonSourceDirectory = getNormalizedAbsolutePath(options.rootDir, currentDirectory);
}
else if (options.composite) {
commonSourceDirectory = getDirectoryPath(normalizeSlashes(options.configFilePath)
}
else {
commonSourceDirectory = computeCommonSourceDirectory(emittedFiles);
}There was a problem hiding this comment.
Changed to something similar
src/compiler/program.ts
Outdated
| if (commonSourceDirectory === undefined) { | ||
| const emittedFiles = filter(files, file => sourceFileMayBeEmitted(file, options, isSourceFileFromExternalLibrary)); | ||
| if (options.rootDir && checkSourceFilesBelongToPath(emittedFiles, options.rootDir)) { | ||
| if (options.composite) { |
There was a problem hiding this comment.
should we check that we have a configFilePath? since if you have niether, normalizeSlashes will get undefined.
src/compiler/program.ts
Outdated
| }; | ||
| } | ||
|
|
||
| function getPrependNodes(): PrependNode[] { |
There was a problem hiding this comment.
ReadonlyArray<PrependNode>?
src/compiler/program.ts
Outdated
| walkProjectReferenceGraph(host, options, (_file, proj, opts) => { | ||
| if (opts.prepend) { | ||
| const dtsFilename = changeExtension(proj.outFile, ".d.ts"); | ||
| const js = host.readFile(proj.outFile) || `/* Input file ${proj.outFile} was missing */\r\n`; |
There was a problem hiding this comment.
if the file does not exist, should we error?
There was a problem hiding this comment.
Added an error upstream
| createDiagnosticForOptionName(Diagnostics.Option_paths_cannot_be_used_without_specifying_baseUrl_option, "paths"); | ||
| } | ||
|
|
||
| if (options.composite) { |
There was a problem hiding this comment.
it should be an error not to have --rootDir or a config file with composite as well. right?
There was a problem hiding this comment.
composite is config only, so it's not an error to not have rootDir - composite just changes how it behaves
src/compiler/program.ts
Outdated
| export function walkProjectReferenceGraph(host: CompilerHost, rootOptions: CompilerOptions, | ||
| callback: (resolvedFile: string, referencedProject: CompilerOptions, settings: ProjectReference) => void, | ||
| errorCallback?: (failedLocation: string) => void) { | ||
| if (rootOptions.references === undefined) return; |
There was a problem hiding this comment.
we already check this in getProjectReferenceFileNames, no need to redo it.
src/compiler/program.ts
Outdated
| errorCallback?: (failedLocation: string) => void) { | ||
| if (rootOptions.references === undefined) return; | ||
|
|
||
| const references = getProjectReferenceFileNames(host, rootOptions); |
There was a problem hiding this comment.
we are going to call resolveProjectReferencePath on every entry a few lines after this, why do we need this pre pass?
src/compiler/program.ts
Outdated
| } | ||
| const referenceJsonSource = parseJsonText(refPath, text); | ||
| const cmdLine = parseJsonSourceFileConfigFileContent(referenceJsonSource, configHost, getDirectoryPath(refPath), /*existingOptions*/ undefined, refPath); | ||
| cmdLine.options.configFilePath = refPath; |
There was a problem hiding this comment.
so why not pass the filepath to parseJsonSourceFileConfigFileContent instead?
src/compiler/program.ts
Outdated
| }; | ||
| } | ||
|
|
||
| export function getProjectReferenceFileNames(host: CompilerHost, rootOptions: CompilerOptions): string[] | undefined { |
There was a problem hiding this comment.
do we even need this function? why not just expose walkProjectReferenceGraph, seems more efficient for most scenarios, and can decide what to do on missing references.
src/compiler/program.ts
Outdated
| return refPath; | ||
| } | ||
|
|
||
| export function walkProjectReferenceGraph(host: CompilerHost, rootOptions: CompilerOptions, |
There was a problem hiding this comment.
for consistency with similar routines in the code base i would call this forEachReference or forEachProjectReference
src/compiler/program.ts
Outdated
| // Note: We don't enumerate the input files to try to find corresponding .d.ts | ||
| // files because we can't know from the outside whether they're modules or not | ||
| if (referencedProject.outFile) { | ||
| const outFile = combinePaths(referencedProject.outDir, changeExtension(referencedProject.outFile, ".d.ts")); |
There was a problem hiding this comment.
the outFile is not relative to the outDir. it is relative to the tsconfig.json file.. maybe i do not understand the rational here.
There was a problem hiding this comment.
it can also depend on declarationDir.
weswigham
left a comment
There was a problem hiding this comment.
Mostly looks like it needs to handle declarationDir all over (probably needs tests for that, too) - the rest is mostly just suggestions for improvements/opinions.
src/compiler/commandLineParser.ts
Outdated
| category: Diagnostics.Module_Resolution_Options, | ||
| description: Diagnostics.Type_declaration_files_to_be_included_in_compilation | ||
| }, | ||
| { |
There was a problem hiding this comment.
Additionally, having a references list implies composite, right? Can we combine the two, eg composite: [ /**/ ] for stuff with dependencies but composite: true or composite: [] for leaf nodes?
Edit from @RyanCavanaugh as I can't reply to this (???) - recycling bits:
References does not imply composite. Leaf-node projects may choose to have references without being composite (for example, it makes no sense to be forced to emit tsc.d.ts, but tsc's tsconfig reasonably has references to checker, etc.).
src/compiler/declarationEmitter.ts
Outdated
| @@ -0,0 +1,2058 @@ | |||
| /// <reference path="checker.ts"/> | |||
src/compiler/diagnosticMessages.json
Outdated
| "category": "Message", | ||
| "code": 6300 | ||
| }, | ||
| "Referenced project '{0}' must have 'declaration': true": { |
There was a problem hiding this comment.
Or itself be composite (which implies this)? Should be mentioned in the error.
There was a problem hiding this comment.
Removed this since it's vestigial from before we put everything under composite
src/compiler/diagnosticMessages.json
Outdated
| "category": "Error", | ||
| "code": 6202 | ||
| }, | ||
| "Projects may not disable declaration emit.": { |
There was a problem hiding this comment.
Ehhhhhhh...... I'd rather we just silently ate the declaration: false in this case and output the declarations into a temp folder, so the user never sees them (so as far as they're concerned, it is off).
There was a problem hiding this comment.
We'd have to figure out what that would look like. tsc doesn't have an arbitrary other-place to drop things to. e.g. What happens if you have 2 checkouts of the same repo and do that - how do we tell them apart? Mirror the entire path from the root into your temp folder? Unlikely to work in Windows, at least, given path length limitations
There was a problem hiding this comment.
A dotfolder in the root of the compilation output, just like the bundle info?
src/compiler/emitter.ts
Outdated
| const declarationFilePath = (forceDtsPaths || options.declaration) ? removeFileExtension(jsFilePath) + Extension.Dts : undefined; | ||
| const declarationMapPath = getAreDeclarationMapsEnabled(options) ? declarationFilePath + ".map" : undefined; | ||
| return { jsFilePath, sourceMapFilePath, declarationFilePath, declarationMapPath }; | ||
| const bundleInfoPath = options.references && jsFilePath && (removeFileExtension(jsFilePath) + ".bundle_info"); |
There was a problem hiding this comment.
Style: .tsbundle or maybe .tsbundledata instead of .bundle_info - because 1. I don't think people will know where a .bundle_info came from (and may delete it), and 2. I dislike underscore in extensions.
Actually - it might make sense to output into a .tscache folder adjacent to the output (and then the file can be named whatever), so the folder can be easily blanket-gitignored.
There was a problem hiding this comment.
Changed to .tsbundleinfo
| context += resetEscapeSequence; | ||
| } | ||
|
|
||
| output += formatColorAndReset(relativeFileName, ForegroundColorEscapeSequences.Cyan); |
There was a problem hiding this comment.
I'm guessing merge error or some kind?
| return undefined; | ||
| } | ||
| if (normalized.indexOf(k) === 0) { | ||
| result = changeExtension(fileName.replace(k, v), ".d.ts"); |
There was a problem hiding this comment.
This seems ignorant of the target project's declarationDir setting - that seems important.
src/compiler/program.ts
Outdated
| const nodes: PrependNode[] = []; | ||
| walkProjectReferenceGraph(host, options, (_file, proj, opts) => { | ||
| if (opts.prepend) { | ||
| const dtsFilename = changeExtension(proj.outFile, ".d.ts"); |
There was a problem hiding this comment.
This needs to account for declarationDir, and/or outDir, right?
There was a problem hiding this comment.
outFile and declarationDir cannot be combined; ref #12405
outFile with outDir - doesn't matter; the output .d.ts file is dropped next to the output .js file
Need an error for a prepended reference that doesn't use outFile, though
src/compiler/program.ts
Outdated
|
|
||
| function createMapping(resolvedFile: string, referencedProject: CompilerOptions, reference: ProjectReference) { | ||
| const rootDir = normalizePath(referencedProject.rootDir || getDirectoryPath(resolvedFile)); | ||
| result.set(rootDir, referencedProject.outDir); |
src/compiler/program.ts
Outdated
| Debug.assert(!!missingFilePaths); | ||
|
|
||
| // List of collected files is complete; validate exhautiveness if this is a project with a file list | ||
| if (options.composite && rootNames.length < files.length) { |
There was a problem hiding this comment.
You might be able to relax/remove this check by using preProcessFile (from preProcess.ts) from the services layer on the importing side to get the files a given file depends on (recursively) without doing a full parse (just a scan), for the most part. Still definitely a good practice to explicitly list all your files, but that could reduce friction in adopting the composite flag.
There was a problem hiding this comment.
When I did the dogfooding conversion, I just used --listFiles to fix the files list manually - seemed to be good enough.
There was a problem hiding this comment.
Any projects that depend on node modules for declarations and/or have a few hundred files I would think may not enjoy managing a list themselves - especially as the exact files in their modules shift under then.
There was a problem hiding this comment.
.d.ts files are not subject to this restriction
There was a problem hiding this comment.
If you have a few hundred files, then your tsconfig would almost certainly be globbing. I'd look askance at anyone not globbing in that situation - detecting an orphaned .ts file that didn't get imported by anyone would be a big pit of failure.
src/compiler/program.ts
Outdated
|
|
||
| function createMapping(resolvedFile: string, referencedProject: CompilerOptions, reference: ProjectReference) { | ||
| const rootDir = normalizePath(referencedProject.rootDir || getDirectoryPath(resolvedFile)); | ||
| result.set(rootDir, referencedProject.outDir); |
There was a problem hiding this comment.
outDir can be empty. do we want to add a mapping to undefined?
| return undefined; | ||
| } | ||
| if (normalized.indexOf(k) === 0) { | ||
| result = changeExtension(fileName.replace(k, v), ".d.ts"); |
There was a problem hiding this comment.
we do have a function to get the declaration file path, can we use that instead?
There was a problem hiding this comment.
getDeclarationEmitOutputFilePath is what i was referring to.
There was a problem hiding this comment.
That function accepts a SourceFile, which we definitely don't have
There was a problem hiding this comment.
But it would be easy to modify since only sourceFile.fileName is used in there anyways.
src/compiler/program.ts
Outdated
|
|
||
| const referencedProjectOutFiles: Path[] = []; | ||
| const projectReferenceRedirects = createProjectReferenceRedirects(options); | ||
| checkProjectReferenceGraph(); |
There was a problem hiding this comment.
can we do this in verifyCompilerOptions instead.
src/compiler/commandLineParser.ts
Outdated
| } | ||
| }, | ||
| { | ||
| name: "projects", |
src/compiler/program.ts
Outdated
| }; | ||
| } | ||
|
|
||
| function getPrependNodes(): PrependNode[] { |
|
@weswigham @mhegazy I believe I've addressed all the comments (as well as offline discussion) - please have a 👀 |
weswigham
left a comment
There was a problem hiding this comment.
Some quick fixes to maybe look at/style comments, but otherwise looks OK.
src/compiler/program.ts
Outdated
|
|
||
| if (options.composite) { | ||
| if (options.declaration === false) { | ||
| createDiagnosticForOptionName(Diagnostics.Projects_may_not_disable_declaration_emit, "declaration"); |
There was a problem hiding this comment.
Projects -> Composite projects?
src/compiler/program.ts
Outdated
|
|
||
| function parseConfigHostFromCompilerHost(host: CompilerHost): ParseConfigFileHost { | ||
| return { | ||
| fileExists: host.fileExists, |
There was a problem hiding this comment.
fileExists(f): f => host.fileExists(f)in case the host uses this.
src/compiler/program.ts
Outdated
| return { | ||
| fileExists: host.fileExists, | ||
| readDirectory: () => [], | ||
| readFile: host.readFile, |
There was a problem hiding this comment.
Should be an arrow in case the host uses this.
src/compiler/program.ts
Outdated
| readDirectory: () => [], | ||
| readFile: host.readFile, | ||
| useCaseSensitiveFileNames: host.useCaseSensitiveFileNames(), | ||
| getCurrentDirectory: host.getCurrentDirectory, |
There was a problem hiding this comment.
Should be an arrow in case the host uses this.
| } | ||
|
|
||
| function transformBundle(node: Bundle) { | ||
| return createBundle(node.sourceFiles.map(transformSourceFile), node.prepends); |
There was a problem hiding this comment.
Maybe map(node.sourceFiles, transformSourceFile) instead of node.sourceFiles.map(transformSourceFile) in case someone decides to do something crazy like an emit with only unparsed sources.
| ES2017 = 4, | ||
| ES2018 = 5, | ||
| ESNext = 6, | ||
| JSON = 100, |
There was a problem hiding this comment.
Seemed like we should keep the ES versions adjacent 🤷♂️
| performance.mark("beforeParse"); | ||
| const result = Parser.parseSourceFile(fileName, sourceText, languageVersion, /*syntaxCursor*/ undefined, setParentNodes, scriptKind); | ||
| let result: SourceFile; | ||
| if (languageVersion === ScriptTarget.JSON) { |
There was a problem hiding this comment.
There's already a ScriptKind.JSON, and there's already a scriptKind parameter - maybe this should just use that?
There was a problem hiding this comment.
The relevant incoming call stack here is via host.getSourceFile, which only accepts ScriptTarget rather than ScriptKind
There was a problem hiding this comment.
I think you want to handle it like in https://github.com/Microsoft/TypeScript/pull/22167/files#diff-ead24f0f0f59c0ea9c1c511052e8884bL671
There was a problem hiding this comment.
@sheetalkamat that seems like the wrong place. Parser exposes parseSourceFile and parseJsonText separately, if someone wants to get back a JSON file they can call parseJsonText instead instead of having parseSourceFile reroute it
| @@ -699,12 +699,14 @@ namespace ts.server { | |||
| } | |||
|
|
|||
| private toFileSpan(fileName: string, textSpan: TextSpan, project: Project): protocol.FileSpan { | |||
There was a problem hiding this comment.
Are there other bits relying on script infos for line mapping that also need to be swapped?
|
@RyanCavanaugh I still owe you feedback on this. I should be able to submit comments by tomorrow. |
|
|
||
| const literalFiles = arrayFrom(literalFileMap.values()); | ||
| const wildcardFiles = arrayFrom(wildcardFileMap.values()); | ||
| const projectReferences = spec.referencesSpecs && spec.referencesSpecs.map((r): ProjectReference => { |
There was a problem hiding this comment.
Why do you need this here? This would get updated only when there is update to tsconfig.json. So you shouldnt want this in here?
There was a problem hiding this comment.
I don't understand the comment - can you expand on that?
| for (const ref of raw.references) { | ||
| if (typeof ref.path !== "string") { | ||
| createCompilerDiagnosticOnlyIfJson(Diagnostics.Compiler_option_0_requires_a_value_of_type_1, "reference.path", "string"); | ||
| } |
There was a problem hiding this comment.
this wont show up error at all in JSON file (with new api) do you have test case to verify this.
There was a problem hiding this comment.
I wasn't sure how to test this. This call seems to match all the others in the file - what is the new API / what would cause this to not show up?
src/compiler/program.ts
Outdated
| } | ||
|
|
||
| // Check if any referenced project tsconfig files are different | ||
| if (createProgramOptions.projectReferences) { |
There was a problem hiding this comment.
just projectReferences instead ?
src/compiler/program.ts
Outdated
|
|
||
| // Check if any referenced project tsconfig files are different | ||
| if (createProgramOptions.projectReferences) { | ||
| for (let i = 0; i < createProgramOptions.projectReferences.length; i++) { |
There was a problem hiding this comment.
Need a check when !projectReferences and oldProgram.projectReferences ?
src/compiler/program.ts
Outdated
| // Check if any referenced project tsconfig files are different | ||
| if (createProgramOptions.projectReferences) { | ||
| for (let i = 0; i < createProgramOptions.projectReferences.length; i++) { | ||
| const oldRef = resolvedProjectReferences[i]; |
There was a problem hiding this comment.
How is resolvedProjectReferences[i] is an oldRef?
| const refPath = resolveProjectReferencePath(host, ref); | ||
| // An absolute path pointing to the containing directory of the config file | ||
| const basePath = getNormalizedAbsolutePath(getDirectoryPath(refPath), host.getCurrentDirectory()); | ||
| const sourceFile = host.getSourceFile(refPath, ScriptTarget.JSON) as JsonSourceFile; |
There was a problem hiding this comment.
Why not use: getParsedCommandLineOfConfigFile instead ?
There was a problem hiding this comment.
Feedback from Mohamed was that we want to pass around SourceFiles instead of filenames so that in-memory editing of referenced projects would work
| @@ -120,7 +120,7 @@ namespace ts { | |||
| createWatchOfConfigFile(configParseResult, commandLineOptions); | |||
There was a problem hiding this comment.
I am assuming you arent adding this portion in this PR. Can you add a TODO and create a issue tracking that. Thanks
src/compiler/core.ts
Outdated
| /** | ||
| * Normalizes a path including its leading drive letter (if any) | ||
| */ | ||
| export function normalizePathAndRoot(path: string): string { |
There was a problem hiding this comment.
All of the places I can find that reference this immediately call toLowerCase() on the result. I would just replace those calls with calls to normalizePath (or resolvePath) instead.
src/compiler/emitter.ts
Outdated
|
|
||
| for (const prepend of bundle.prepends) { | ||
| print(EmitHint.Unspecified, prepend, /*sourceFile*/ undefined); | ||
| write("\n"); |
src/compiler/emitter.ts
Outdated
| case EmitHint.MappedTypeParameter: return emitMappedTypeParameter(cast(node, isTypeParameterDeclaration)); | ||
| case EmitHint.Unspecified: return pipelineEmitUnspecified(node); | ||
| default: | ||
| return Debug.assertNever(hint, `Unhandled EmitHint: ${(ts as any).EmitHint ? (ts as any).EmitHint[hint] : hint }`); |
There was a problem hiding this comment.
take a look at how formatSyntaxKind works in utilities.ts and possibly add a format function for this.
|
🎉🎉 |
Fixes most of #3469
There are two separable but intertwined concepts present in this PR: Project references, and composite projects.
A project reference is an entry in a
tsconfigfile that points to anothertsconfigfile:A project reference in a tsconfig file performs a redirection when resolving module names. Writing writing:
If a project reference subsumes
../otheror../other/dir, instead of looking formod.ts/mod.tsxfiles in../other/dir, we look for.d.tsfiles in theoutDirof the referenced project.In this PR, project references may not be circular.
Project references may include
prepend: trueinoutFilecompilations. The output of aprepend: trueis prepended to the output of theoutFileof the current project. Additionally, we write a.bundle_infofile indicating where "this" project's JS starts so that tools (seetsbuildbelow) can re-use the downstream project JS if the upstream project JS changes without modifying its .d.ts content.In order to be referenced, a tsconfig file must be a
compositeproject. This is done by setting"composite": true(including possibly in a base tsconfig file). Composite projects are subject to additional constraints:rootDiris inferred differently if not specifiedThese rules define an additional property of composite projects: They can be trivially determined to be up-to-date or not, because the set of input files and output files can be determined without reading any source files from disk.
tsbuild leverages these properties to enable extremely fast rebuilds of large solutions. The projectReferencesDogfood branch demonstrates what this looks like for an
outFilecompilation.