Skip to content

Proposal: Drop the traceback from the module-level backend error objects.#112

Merged
cpburnz merged 1 commit intocpburnz:masterfrom
yilei:remove-traceback
Mar 18, 2026
Merged

Proposal: Drop the traceback from the module-level backend error objects.#112
cpburnz merged 1 commit intocpburnz:masterfrom
yilei:remove-traceback

Conversation

@yilei
Copy link
Contributor

@yilei yilei commented Mar 17, 2026

Why? These are module-level objects that will live forever. Normally this is fine. But if we lazily import pathspec in somewhere deep in the Python program (or with the upcoming PEP 810 explicit lazy import in Python 3.15), it will hold the traceback, thus the locals of each frame, in memory forever.

We ran into such case where we might hold GBs of memory because of this edge case.

As far as I can tell, we don't really need the traceback on the exception object here.

What do you think? I can totally understand this seems to be out of the blue optimization for super rare edge cases.

Thanks for your consideration in advance!

@cpburnz
Copy link
Owner

cpburnz commented Mar 18, 2026

It is obscure, but there's no strong benefit of keeping the original traceback. This is with the original traceback:

>>> pathspec.PathSpec.from_lines('gitignore', [], backend='re2')
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    pathspec.PathSpec.from_lines('gitignore', [], backend='re2')
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/pathspec.py", line 261, in from_lines
    return cls(patterns, backend=backend, _test_backend_factory=_test_backend_factory)
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/pathspec.py", line 82, in __init__
    use_backend = self._make_backend(backend, patterns)
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/pathspec.py", line 281, in _make_backend
    return make_pathspec_backend(name, patterns)
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/_backends/agg.py", line 100, in make_pathspec_backend
    return Re2PsBackend(cast(Sequence[RegexPattern], patterns))
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/_backends/re2/pathspec.py", line 58, in __init__
    raise re2_error
  File "/home/caleb/Downloads/venv/lib/python3.14/site-packages/pathspec/_backends/re2/_base.py", line 17, in <module>
    import re2
ModuleNotFoundError: No module named 're2'

And this is using your change to remove the original traceback:

>>> pathspe.PathSpec.from_lines('gitignore', [], backend='re2')
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    pathspec.PathSpec.from_lines('gitignore', [], backend='re2')
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 267, in from_lines
    return cls(patterns, backend=backend, _test_backend_factory=_test_backend_factory)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 82, in __init__
    use_backend = self._make_backend(backend, patterns)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 287, in _make_backend
    return make_pathspec_backend(name, patterns)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/_backends/agg.py", line 100, in make_pathspec_backend
    return Re2PsBackend(cast(Sequence[RegexPattern], patterns))
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/_backends/re2/pathspec.py", line 58, in __init__
    raise re2_error
ModuleNotFoundError: No module named 're2'

I don't think keeping the original traceback adds any important information, particularly because the exception remains a clear ModuleNotFoundError exception. And the odd case of pyre2 being installed isn't any less clear:

>>> pathspec.PathSpec.from_lines('gitignore', [], backend='re2')
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    pathspec.PathSpec.from_lines('gitignore', [], backend='re2')
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 267, in from_lines
    return cls(patterns, backend=backend, _test_backend_factory=_test_backend_factory)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 82, in __init__
    use_backend = self._make_backend(backend, patterns)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/pathspec.py", line 287, in _make_backend
    return make_pathspec_backend(name, patterns)
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/_backends/agg.py", line 100, in make_pathspec_backend
    return Re2PsBackend(cast(Sequence[RegexPattern], patterns))
  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/_backends/re2/pathspec.py", line 58, in __init__
    raise re2_error
AttributeError: module 're2' has no attribute 'Options'

with this missing:

  File "/home/caleb/Personal/Projects/python/python-pathspec/pathspec/_backends/re2/_base.py", line 27, in <module>
    RE2_OPTIONS = re2.Options()
                  ^^^^^^^^^^^

Thanks.

@cpburnz cpburnz merged commit 68ee314 into cpburnz:master Mar 18, 2026
15 of 64 checks passed
@yilei
Copy link
Contributor Author

yilei commented Mar 18, 2026

Thank you for taking a close look at the tracebacks and merging my PR!

@yilei yilei deleted the remove-traceback branch March 18, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants