[stubtest] Check runtime availability of private types not marked @type_check_only

[brianschubert]

Currently, stubtest ignores symbols that are 1) private and 2) not present at runtime. Private names are often used for stub-only constructs (which we don't want stubtest to complain about), but they can also denote actual private runtime types. Without other context, it's often ambiguous which category a private names falls into. For class definitions, we can make the intent explicit by decorating the class with @type_check_only. Typeshed already does this for the most part, but there's still quite a few cases of stub-only private types that aren't marked @type_check_only (especially in older stubs).

Going forward, I think it could a good idea to make stubtest start checking for @type_check_only on private types that are not available at runtime.

A few special cases are needed to avoid false positives on types that cannot be marked with @type_check_only, namely:

• NewType declarations
• NamedTuples and TypedDicts with fields that are not valid identifiers (and therefore need to use the functional syntax)

This PR creates quite a few new errors on typeshed (~100 in stdlib, ~300 in third-party stubs), which we'll probably want to sift through before merging this. So far I've gone through the stdlib hits, all of which seem to be true positives that we'd want to update.

[brianschubert]

python/typeshed#14512 addresses the new stdlib hits in typeshed

thirdy-party stub hits can be seen at https://github.com/brianschubert/typeshed/actions/runs/16697076815/

[brianschubert]

These are the third-party stub hits if anyone is interested in investigating some (I'll be sifting through them as I have time)

Details

$ cat *stubtest* | tr -d '\r' | sed -e 's/\x1b[^m]*m//g' | cut -d' ' -f2- | grep '^error' | sort | uniq
error: aiofiles.threadpool.binary._UnknownAsyncBinaryIO is not present at runtime. Maybe mark it as "@type_check_only"?
error: aiofiles.threadpool.text._UnknownAsyncTextIO is not present at runtime. Maybe mark it as "@type_check_only"?
error: authlib.oauth2.rfc6750.token._TokenGenerator is not present at runtime. Maybe mark it as "@type_check_only"?
error: bleach.callbacks._Callback is not present at runtime. Maybe mark it as "@type_check_only"?
error: bleach.html5lib_shim._Encoding is not present at runtime. Maybe mark it as "@type_check_only"?
error: boltons.setutils._RSub is not present at runtime. Maybe mark it as "@type_check_only"?
error: boltons.typeutils._Sentinel is not present at runtime. Maybe mark it as "@type_check_only"?
error: braintree.util.graphql_client._Error is not present at runtime. Maybe mark it as "@type_check_only"?
error: braintree.util.graphql_client._Extension is not present at runtime. Maybe mark it as "@type_check_only"?
error: braintree.util.graphql_client._Response is not present at runtime. Maybe mark it as "@type_check_only"?
error: braintree.util.graphql_client._ValidationErrors is not present at runtime. Maybe mark it as "@type_check_only"?
error: _cffi_backend._Allocator is not present at runtime. Maybe mark it as "@type_check_only"?
error: click_spinner._Stream is not present at runtime. Maybe mark it as "@type_check_only"?
error: croniter.croniter._AllIter is not present at runtime. Maybe mark it as "@type_check_only"?
error: dateparser._Settings is not present at runtime. Maybe mark it as "@type_check_only"?
error: dateutil.tz._common._DatetimeWithFold is not present at runtime. Maybe mark it as "@type_check_only"?
error: dateutil.tz.tz._GetTZ is not present at runtime. Maybe mark it as "@type_check_only"?
error: dateutil.tz.tz._ICalReader is not present at runtime. Maybe mark it as "@type_check_only"?
error: dateutil.tz.tz._TZFileReader is not present at runtime. Maybe mark it as "@type_check_only"?
error: defusedxml.lxml._ElementBase is not present at runtime. Maybe mark it as "@type_check_only"?
error: defusedxml.xmlrpc._Readable is not present at runtime. Maybe mark it as "@type_check_only"?
error: dockerfile_parse.parser._InstructionDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: docker.models.containers._RestartPolicy is not present at runtime. Maybe mark it as "@type_check_only"?
error: docker.utils.proxy._Environment is not present at runtime. Maybe mark it as "@type_check_only"?
error: docker.utils.proxy._ProxyConfigDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: docutils.nodes._DomModule is not present at runtime. Maybe mark it as "@type_check_only"?
error: fanstatic.injector._InjectorPluginOptions is not present at runtime. Maybe mark it as "@type_check_only"?
error: fanstatic.injector._NeededResourcesConfig is not present at runtime. Maybe mark it as "@type_check_only"?
error: fanstatic.injector._TopBottomInjectorPluginOptions is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_cors.core._Options is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_migrate._SupportsWriteAndFlush is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio._EngineIOServerConfig is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio._ExceptionHandlerDecorator is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio._HandlerDecorator is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio.namespace._BaseNamespace is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio._SocketIOKwargs is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio._SocketIOServerOptions is not present at runtime. Maybe mark it as "@type_check_only"?
error: flask_socketio.test_client._Packet is not present at runtime. Maybe mark it as "@type_check_only"?
error: fpdf.encryption._SupportsGetItem is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb._FrameFilter is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb._InferiorCallEvent is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb._PrettyPrinter is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb.types._TypePrinter is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb.types._TypeRecognizer is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb._Window is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb.xmethod._SimpleWorkerMethod is not present at runtime. Maybe mark it as "@type_check_only"?
error: gdb.xmethod._XMethod is not present at runtime. Maybe mark it as "@type_check_only"?
error: geopandas.io.file._Schema is not present at runtime. Maybe mark it as "@type_check_only"?
error: geopandas.io.sql._SqlalchemyConnectionLike is not present at runtime. Maybe mark it as "@type_check_only"?
error: geopandas.io.sql._SqlalchemyEngineLike is not present at runtime. Maybe mark it as "@type_check_only"?
error: geopandas.io.sql._SqlAlchemyEventTarget is not present at runtime. Maybe mark it as "@type_check_only"?
error: geopandas.io.sql._SqlalchemyTransactionLike is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.baseserver._SpawnFunc is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent._config._SettingDescriptor is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.events._PeriodicMonitorThread is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.event._ValueSource is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent._ffi.loop._SupportsHandleError is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.hub._DefaultReturnProperty is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent._hub_primitives._Waitable is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.pywsgi._LogOutputStream is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.resolver.ares._ChannelArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.server._SSLArguments is not present at runtime. Maybe mark it as "@type_check_only"?
error: gevent.timeout._HasSeconds is not present at runtime. Maybe mark it as "@type_check_only"?
error: google.protobuf.internal.containers._ValueChecker is not present at runtime. Maybe mark it as "@type_check_only"?
error: google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper is not present at runtime. Maybe mark it as "@type_check_only"?
error: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor is not present at runtime. Maybe mark it as "@type_check_only"?
error: google.protobuf.internal.well_known_types._FieldMaskTree is not present at runtime. Maybe mark it as "@type_check_only"?
error: greenlet._greenlet._ParentDescriptor is not present at runtime. Maybe mark it as "@type_check_only"?
error: html5lib._inputstream._Encoding is not present at runtime. Maybe mark it as "@type_check_only"?
error: hvac.adapters._GenericRawAdapter is not present at runtime. Maybe mark it as "@type_check_only"?
error: hvac.utils._DeprecateProperty is not present at runtime. Maybe mark it as "@type_check_only"?
error: icalendar.prop._vType is not present at runtime. Maybe mark it as "@type_check_only"?
error: jack._CBufferType is not present at runtime. Maybe mark it as "@type_check_only"?
error: jack._JackPositionT is not present at runtime. Maybe mark it as "@type_check_only"?
error: jmespath.ast._ASTNode is not present at runtime. Maybe mark it as "@type_check_only"?
error: jmespath.functions._Signature is not present at runtime. Maybe mark it as "@type_check_only"?
error: jmespath.lexer._LexerTokenizeResult is not present at runtime. Maybe mark it as "@type_check_only"?
error: jmespath.visitor._TreeNode is not present at runtime. Maybe mark it as "@type_check_only"?
error: jsonschema.validators._Validator is not present at runtime. Maybe mark it as "@type_check_only"?
error: markdown.extensions.toc._FlatTocToken is not present at runtime. Maybe mark it as "@type_check_only"?
error: markdown.extensions.toc._TocToken is not present at runtime. Maybe mark it as "@type_check_only"?
error: markdown.util._TagData is not present at runtime. Maybe mark it as "@type_check_only"?
error: mock.mock._patcher is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResulHostUptime is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultHostNames is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultHostPort is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._Result is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultNampInfoProtocol is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultNampStats is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultNmapInfo is not present at runtime. Maybe mark it as "@type_check_only"?
error: nmap.nmap._ResultNmap is not present at runtime. Maybe mark it as "@type_check_only"?
error: oauthlib.oauth2.rfc6749.request_validator._AuthorizationCode is not present at runtime. Maybe mark it as "@type_check_only"?
error: oauthlib.oauth2.rfc6749.request_validator._BearerToken is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl._Decodable is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.descriptors.sequence._NestedSequenceToTreeObj is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.descriptors.sequence._SupportsFromTree is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.descriptors.sequence._SupportsToTree is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.descriptors.serialisable._ChildSerialisableTreeElement is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.descriptors.serialisable._SerialisableTreeElement is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.worksheet.datavalidation._HasCoordinate is not present at runtime. Maybe mark it as "@type_check_only"?
error: openpyxl.worksheet._writer._SupportsCloseAndWrite is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.agent._AgentProxy is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.client._TransportFactory is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.hostkeys._SubDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.message._SupportsAsBytes is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.pipe._BasePipe is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.pipe._Pipe is not present at runtime. Maybe mark it as "@type_check_only"?
error: paramiko.transport._KexEngine is not present at runtime. Maybe mark it as "@type_check_only"?
error: pexpect.spawnbase._Logfile is not present at runtime. Maybe mark it as "@type_check_only"?
error: pika.credentials._Credentials is not present at runtime. Maybe mark it as "@type_check_only"?
error: psycopg2._psycopg._ISQLQuoteProto is not present at runtime. Maybe mark it as "@type_check_only"?
error: psycopg2._psycopg._SupportsReadAndReadline is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._AnnotationG is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._Annotation is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._Category is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._Dataset is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.cocoeval._EvaluationResult is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.cocoeval._ImageEvaluationResult is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._Image is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools.coco._RLE is not present at runtime. Maybe mark it as "@type_check_only"?
error: pycocotools._EncodedRLE is not present at runtime. Maybe mark it as "@type_check_only"?
error: pygments.style._StyleDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: PyInstaller.utils.hooks.conda._RawDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: pynput._util._RESOLUTIONS is not present at runtime. Maybe mark it as "@type_check_only"?
error: pytz._UTCclass is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.barcode.qr._isLevel is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.barcode.qr._isUnicodeOrQRList is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._AllPathKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._DefinePathKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._DrawingKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._GroupKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._isStrokeDashArray is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._LineShapeKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._PathKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._SolidShapeKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._StringKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.graphics.shapes._WedgeKwArgs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.lib.colors._cssParse is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.lib.colors._toColor is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.lib.rparsexml._smartDecode is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.lib.utils._UNSET_ is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.doctemplate._CanvasMaker is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.flowables._StyledFlowableFactory is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.para._LineOpHandler is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.para._SimpleStyleKwargs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.para._SupportsWidthAndExecute is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.tableofcontents._SimpleIndexKwargs is not present at runtime. Maybe mark it as "@type_check_only"?
error: reportlab.platypus.tableofcontents._TableOfContentsKwargs is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.adapters._HostParams is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.adapters._PoolKwargs is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._ImplementationDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._InfoDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._OptionalVersionDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._PlatformDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._PyOpenSSLDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.help._VersionDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests_oauthlib.oauth1_session._TokenDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests_oauthlib.oauth2_session._AccessTokenResponseHook is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests_oauthlib.oauth2_session._ComplianceHooks is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests_oauthlib.oauth2_session._ProtectedRequestHook is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests_oauthlib.oauth2_session._RefreshTokenResponseHook is not present at runtime. Maybe mark it as "@type_check_only"?
error: requests.sessions._Settings is not present at runtime. Maybe mark it as "@type_check_only"?
error: seaborn.distributions._Fit is not present at runtime. Maybe mark it as "@type_check_only"?
error: seaborn.external.kde._GaussianKDELike is not present at runtime. Maybe mark it as "@type_check_only"?
error: serial.urlhandler.protocol_spy._Formatter is not present at runtime. Maybe mark it as "@type_check_only"?
error: setuptools.command.editable_wheel._DebuggingTips is not present at runtime. Maybe mark it as "@type_check_only"?
error: setuptools.config.pyprojecttoml._BetaConfiguration is not present at runtime. Maybe mark it as "@type_check_only"?
error: setuptools.config.pyprojecttoml._InvalidFile is not present at runtime. Maybe mark it as "@type_check_only"?
error: singledispatch._SingleDispatchCallable is not present at runtime. Maybe mark it as "@type_check_only"?
error: toposort._SupportsItemsAndLen is not present at runtime. Maybe mark it as "@type_check_only"?
error: tqdm.dask._Callback is not present at runtime. Maybe mark it as "@type_check_only"?
error: tqdm.rich._ProgressColumn is not present at runtime. Maybe mark it as "@type_check_only"?
error: tqdm.utils._Has__Comparable is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._ChameleonTranslate is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._Pluralizer is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._PluralizerPolicy is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._TranslationStringFactory is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._Translator is not present at runtime. Maybe mark it as "@type_check_only"?
error: translationstring._TranslatorPolicy is not present at runtime. Maybe mark it as "@type_check_only"?
error: uwsgi._RPCCallable is not present at runtime. Maybe mark it as "@type_check_only"?
error: watchpoints.watch._TraceFunc is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.acceptparse._SupportsStr is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.cookies._Serializer is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.dec._unbound_wsgify is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.exc._JSONFormatter is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.multidict._SupportsItemsWithIterableResult is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.request._RequestCacheControlDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.request._SupportsReadAndNoArgReadline is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.response._ResponseCacheControlDict is not present at runtime. Maybe mark it as "@type_check_only"?
error: webob.response._ResponseCacheExpires is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.fields.core._Validator is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.fields.core._Widget is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.form._UnboundFields is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.i18n._SupportsUgettextAndUngettext is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.meta._MultiDictLikeBase is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.meta._MultiDictLikeWithGetall is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.meta._MultiDictLikeWithGetlist is not present at runtime. Maybe mark it as "@type_check_only"?
error: wtforms.meta._SupportsGettextAndNgettext is not present at runtime. Maybe mark it as "@type_check_only"?
error: wurlitzer._Stream is not present at runtime. Maybe mark it as "@type_check_only"?
error: Xlib.display._ResourceBaseClassesType is not present at runtime. Maybe mark it as "@type_check_only"?
error: Xlib.rdb._SupportsComparisons is not present at runtime. Maybe mark it as "@type_check_only"?
error: yaml.emitter._WriteStream is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.feedback._Feedback is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.matching._DM is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.matching._DMY is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.matching._Match is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn._Result is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.scoring._GuessesResult is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.time_estimates._CrackTimesDisplay is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.time_estimates._CrackTimeSeconds is not present at runtime. Maybe mark it as "@type_check_only"?
error: zxcvbn.time_estimates._TimeEstimate is not present at runtime. Maybe mark it as "@type_check_only"?

[brianschubert]

Marking this as ready for review.

There's still third-party stub errors in typeshed that will need to be addressed, but there should be plenty of time to look into those by the time this reaches typeshed. So far the ones I've checked have been true positives (edit: the new third-party errors have been fixed in python/typeshed#14545)

[Avasam]

Thanks for taking a shot at this ! It's something I've been wanting for a while as well. This works towards #15146