build: enable line tables in release builds so backtraces show file:line (#103)#546
Open
LeSingh1 wants to merge 1 commit into
Open
build: enable line tables in release builds so backtraces show file:line (#103)#546LeSingh1 wants to merge 1 commit into
LeSingh1 wants to merge 1 commit into
Conversation
Stack traces from release-built tiktoken wheels do not include file or line numbers, which makes debugging panics that surface through the Python extension harder than it needs to be (cf. openai#103). Set `debug = "line-tables-only"` for the release profile. This adds just enough DWARF for backtraces to resolve file:line, without the binary-size cost of full `debug = true`. Optimization level is unaffected, so the original concern that motivated stripping debug info — runtime performance — is preserved (the benchmark in openai#103 showed identical throughput before/after enabling debug).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #103.
Release builds of the tiktoken Rust extension strip all debug info, so panic stack traces surface in Python without any file or line numbers. Adding
debug = "line-tables-only"to the release profile keeps just enough DWARF to resolve backtraces to file:line, without the binary-size cost of fulldebug = true.Why
line-tables-onlyand notdebug = true?The reporter's original suggestion was
debug = true, and their benchmark in the issue showed no runtime impact. Both options give you the line numbers.line-tables-only(stable since Rust 1.69) is a strict subset that drops variable / type information while keeping.debug_line, so:function_name (path/file.rs:42)✅debug = true.debug_*sections, which aren't loaded into memory at runtime)Happy to switch to
debug = trueif maintainers prefer it for parity with the original issue.MSRV check
line-tables-onlywas stabilized in Rust 1.69 (May 2023).Cargo.tomlalready declaresedition = "2024", which requires Rust 1.85+, so this is well within the existing MSRV.Verification
cargo check --release --all-featurespasses (only pre-existing pyo3 deprecation warning).