Add comments parsing flag to ast.parse

[NeilGirdhar]

Feature or enhancement

Add a flag to ast.parse that signals it to parse comments for directives given to various static analysis tools like PyRight, MyPy, Pylint, etc.

Pitch

Currently, ast.parse produces directives for type: ignore only, which all type checkers respond to. PyRight reacts to its own special pyright: ignore flag to disambiguate type errors that are specific to it. MyPy cannot easily implement a similar flag since it depends on ast.parse. Yet a mypy: ignore directive is needed.

A concrete interface is proposed here by @jab.

Previous discussion

Discussion here. Guido suggests a passing a list of namespaces the caller is interested in. This would make the function output easier to parse, and possibly more efficient.

[NeilGirdhar]

It would be nice to also support associating comment directives from preceding otherwise empty lines, e.g., like this.

[srittau]

Personally, I like the suggestion by @jhance in this comment better: Add a flag to "type: ignore" to indicate which type checker(s) it applies to. This would need no ast changes, only an amendment of the typing spec. The latter chapter need updating anyway, since it doesn't reflect the current reality of "# type: ignore" comments.

[NeilGirdhar]

@srittau Right, the downside of that approach is that some other linter or tool may want to parse something else. Also, pyright and Ruff will probably never change what they respond to (and while they don't use AST, some tool may want to parse their directives?).

I guess I'm an idealist, and I want a perfect solution, but I agree that your suggestion is easier to get to from where we are. Also, your solution doesn't preclude one day implementing the "perfect solution", which is nice.

The latter chapter need updating anyway, since it doesn't reflect the current reality of "# type: ignore" comments.

Great point.

[wyattscarpenter]

I think it would be nice to have this feature.

I also think it would probably be easier and more extensible if ast.parse exposed all comments in its results. Then all libraries hoping to read their directives in comments could do so on their end. This would allow flexibility; for instance mypy could choose to do its directives in either way. This would also match the behavior of https://docs.python.org/3/library/tokenize.html, which is intended to match the behavior of ast.parser except it also exposes COMMENT tokens.

All three of these mechanisms (including the current type: ignore mechanism, which already exists) could co-exist. Of them, I would say exposing all comments is the most important. Maybe there's some reason not to do that which I don't know of.