Pyinstaller command strip away important traceback details

[FuckingToasters]

Note: This Discussion originally started on dashingsoft/pyarmor#1789

Hello, so i used following pyinstaller command to build te exe:

:: cd D:\workspace\joinify
C:\Users\Infinimonster\venv\Scripts\activate && pyinstaller --clean --onefile --icon=icon.ico --add-binary=c:\users\infinimonster\appdata\local\programs\python\python39\lib\site-packages\tls_client\dependencies\tls-client-64.dll:tls_client\dependencies Joinify.py

I would like to show the same error amount which shows when running the Joinify.py file directly.

I use the rich libary to save beautified Traceback info as HTML File but the output only contain the function name when running the comopiled executable. Not even the file etc.

Note that i also use a custom Joinify.spec file with the content listed below:

# -*- mode: python ; coding: utf-8 -*-


a = Analysis(
    ['Joinify.py'],
    pathex=[],
    binaries=[('c:\\users\\infinimonster\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\tls_client\\dependencies\\tls-client-64.dll', 'tls_client\\dependencies')],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
    optimize=0,
)
pyz = PYZ(a.pure)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.datas,
    [],
    name='Joinify',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
    icon=['icon.ico'],
)

I would like to let it show every error detail and not only the function name and line count (the line count is a few lines off too plus it don't show the file it occured in.)

[rokm]

I suspect that if you want detailed tracebacks, you need source .py files, and PyInstaller does not collect those by default.

As for the lines being off, I've never noticed that with "built-in" tracebacks. So it might be the artifact of the traceback library you are using (perhaps caused by the lack of source .py files).

[rokm]

Are we talking about the minimal example you gave earlier, or is this your full application again? Can you confirm that your minimal example now works as expected when frozen, or not?

I built the full application. with the minimal example i only took out some not neccecary important stuff for error replication.

Yes, but the thing is, if the minimal example doesn't work for you, then there's no reason to believe that the full application will. So we should first confirm that minimal example works, and then move on to the full application...

[FuckingToasters]

Are we talking about the minimal example you gave earlier, or is this your full application again? Can you confirm that your minimal example now works as expected when frozen, or not?

I built the full application. with the minimal example i only took out some not neccecary important stuff for error replication.

Yes, but the thing is, if the minimal example doesn't work for you, then there's no reason to believe that the full application will. So we should first confirm that minimal example works, and then move on to the full application...

I found out something intresting:

Se, the error print correctly with a lot of detail if there are errors which are not captched by the exception (i.e undefined variable) but as soon as everything is defined and then an error happens at the 1/0 part, not much info is given

[rokm]

Is it really printed "correctly", though? It did dump locals from the frames, but there are no lines of code there, because the source cannot be retrieved due to co_filename not being a valid filename. You can further try to instrument the rendering part of the rich.traceback code (around here) to see what exactly is happening.

[FuckingToasters]

My rich libary modifications:

            suppressed = any(frame_filename.startswith(path) for path in self.suppress)

            # Debugging: Print out the frame details
            print(f"Debug: frame.filename={frame.filename}, frame.lineno={frame.lineno}, frame.name={frame.name}")

            if os.path.exists(frame.filename):
                text = Text.assemble(
                    path_highlighter(Text(frame.filename, style="pygments.string")),
                    (":", "pygments.text"),
                    (str(frame.lineno), "pygments.number"),
                    " in ",
                    (frame.name, "pygments.function"),
                    style="pygments.text",
                )
            else:
                # Check if co_filename is valid and attempt to use it if frame has code attribute
                frame_code_filename = getattr(frame, 'code', None)
                if frame_code_filename and os.path.exists(frame_code_filename.co_filename):
                    text = Text.assemble(
                        path_highlighter(Text(frame_code_filename.co_filename, style="pygments.string")),
                        (":", "pygments.text"),
                        (str(frame.lineno), "pygments.number"),
                        " in ",
                        (frame.name, "pygments.function"),
                        style="pygments.text",
                    )
                else:
                    text = Text.assemble(
                        "in ",
                        (frame.name, "pygments.function"),
                        (":", "pygments.text"),
                        (str(frame.lineno), "pygments.number"),
                        style="pygments.text",
                    )

            if not frame.filename.startswith("<") and not first:
                yield ""
            yield text
            if frame.filename.startswith("<"):
                # Additional debug information if needed
                print(f"Debug: Invalid or internal frame.filename detected: {frame.filename}")
                yield from render_locals(frame)
                continue

My updated minimal example:

def run_thread(obf, current_dir):
    try:
        a = 1/0 # purposefully cause error for testing

    except Exception as E:
        def pretty_trace(file: bool, tempfile: str):
            file = False # TEMPORARY
            from rich.traceback import Traceback as rtb
            from rich.markdown import Markdown as rmarkdown
            from sys import _getframe
            from types import TracebackType
            tb = None
            depth = 1
            while True:
                try:
                    frame = _getframe(depth)
                    depth += 1
                except ValueError as exc:
                    break
                tb = TracebackType(tb, frame, frame.f_lasti, frame.f_lineno)
            ex = Exception('pretty_trace')
            stack = rtb.extract(type(ex), ex, tb, show_locals=True)
            
            if not file:
                console = rconsole(record=True)
                console.print(rtb(stack, show_locals=True))
            else:
                with open(tempfile, "w", encoding="utf-8") as file:
                    console = rconsole(record=True, file=file)
                    console.print(rtb(stack, show_locals=True))
                    #console.print_exception(show_locals=True)
                console.save_html(tempfile)
                
                with open(tempfile, "r", encoding="utf-8") as file:
                    lines = file.read()
                    #exception_info = utility.get_exception_info(E)           
                    encstr = lines

                with open(tempfile, "w+", encoding="utf-8") as file:
                    file.write(encstr)
                
            #rconsole.print(rtb(stack, show_locals=True))
            
        tempfile = os.path.join(current_dir, "error.html") if current_dir else "error.html"
        pretty_trace(obf, tempfile)
        print(f"Oops, this was unexpected, open a ticket at https://genefit.to/discord and upload {tempfile}")
        #console = rconsole(record=True)
        #console.print(pretty_trace())
        #console.print_exception(show_locals=True)

if __name__ == "__main__":
    import sys,os
    import traceback
    from rich import _IMPORT_CWD
    from rich.console import Console as rconsole
    from rich.traceback import Traceback as rtb
    from rich.traceback import install as rtbinstall; rtbinstall(show_locals=True)
	
    obf = False
    if getattr(sys, 'frozen', False):
        _IMPORT_CWD = sys._MEIPASS
        obf = True
    
    current_dir = os.path.dirname(os.path.abspath(__file__)) if obf is False else None
    
    run_thread(obf, current_dir)

Do you have any clue as to why this is happening and how i could solve it? I tried a lot of things but none of them worked in my favour.

[rokm]

Do you have any clue as to why this is happening and how i could solve it?

I'm not sure what the question here is.

If you are using just PyInstaller, the above sample won't display lines of code (even with --add-data richlib.py:.) because you are not overriding _IMPORT_CWD correctly, as per earlier comments.

And if this is about PyArmor (seeing how entry-point is referred to as <frozen richlib>), I can't really help you with that. As far as I can tell, co_filename precludes the look-up of the sources, which is necessary to display lines of code. I suppose you could hack the filename inference in rich.traceback (parse the module name from <frozen modulename> and turn that into filename), but again, I don't think it makes any sense. You are trying to obfuscate the byte-compiled modules with PyArmor, but having to add unobfuscated source code to the build in order to display those detailed tracebacks fully negates purpose of PyArmor. Then again, you might want to get a second opinion on that from PyArmor folks.