Crash on AST with misordered linenos

[JelleZijlstra]

• Compile 3.11 branch
• ./python.exe -m ensurepip
• ./python.exe -m pip install pytest-cov
• ./python.exe -m pytest

Crashes with Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.

lldb stack trace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00007fff203cd92e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff203fc5bd libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007fff20351406 libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff203507d8 libsystem_c.dylib`__assert_rtn + 314
  * frame #4: 0x000000010027c183 python.exe`write_location_info_long_form.cold.2 at compile.c:7532:5 [opt]
    frame #5: 0x000000010018f469 python.exe`write_location_info_long_form(a=<unavailable>, i=<unavailable>, length=<unavailable>) at compile.c:7532:5 [opt]
    frame #6: 0x000000010018f27d python.exe`write_location_info_entry(a=0x00007ffeefbfe5d0, i=0x000000010388ac98, isize=1) at compile.c:7590:5 [opt]
    frame #7: 0x000000010018d2cd python.exe`assemble_emit_location(a=0x00007ffeefbfe5d0, i=0x000000010388ac98) at compile.c:7605:12 [opt]
    frame #8: 0x000000010017e9fa python.exe`assemble(c=<unavailable>, addNone=<unavailable>) at compile.c:8411:18 [opt]
    frame #9: 0x000000010017d4a3 python.exe`compiler_mod(c=0x00007ffeefbfe6b8, mod=<unavailable>) at compile.c:2203:10 [opt]
    frame #10: 0x000000010017d2b3 python.exe`_PyAST_Compile(mod=0x00000001038852b0, filename=0x000000010654d630, flags=<unavailable>, optimize=<unavailable>, arena=0x00000001064b0100) at compile.c:581:10 [opt]
    frame #11: 0x00000001001634e0 python.exe`builtin_compile_impl(module=<unavailable>, source=0x00000001065a3fc0, filename=0x000000010654d630, mode=<unavailable>, flags=<unavailable>, dont_inherit=1, optimize=-1, feature_version=-1) at bltinmodule.c:818:33 [opt]
    frame #12: 0x0000000100161fb6 python.exe`builtin_compile(module=<unavailable>, args=0x00007ffeefbfe7f0, nargs=<unavailable>, kwnames=<unavailable>) at bltinmodule.c.h:328:20 [opt]
    frame #13: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=<unavailable>, kwnames=0x0000000102c72260) at methodobject.c:443:24 [opt]
    frame #14: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=9223372036854775811, kwnames=0x0000000102c72260) at pycore_call.h:92:11 [opt]
    frame #15: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #16: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6fd70, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #17: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #18: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104737d80, locals=0x0000000000000000, args=0x00007ffeefbfeb80, argcount=3, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #19: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104737d80, stack=0x00007ffeefbfeb80, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #20: 0x0000000100089fd8 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000104737d80, args=0x00007ffeefbfeb80, nargsf=3, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #21: 0x0000000100089644 python.exe`method_vectorcall(method=<unavailable>, args=0x00000001064c9788, nargsf=<unavailable>, kwnames=0x0000000000000000) at classobject.c:89:18 [opt]
    frame #22: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`method_vectorcall at classobject.c:44), callable=0x000000010640d310, tuple=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #23: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x000000010640d310, args=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #24: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #25: 0x0000000100177419 python.exe`do_call_core(tstate=0x0000000100486930, func=0x000000010640d310, callargs=0x00000001064c9770, kwdict=0x0000000000000000, use_tracing=<unavailable>) at ceval.c:0:13 [opt]
    frame #26: 0x00000001001733b0 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f580, throwflag=<unavailable>) at ceval.c:5431:22 [opt]
    frame #27: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #28: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104053490, locals=0x0000000000000000, args=0x00000001064c0a48, argcount=1, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #29: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104053490, stack=0x00000001064c0a48, nargsf=<unavailable>, kwnames=0x00000001064c8f50) at call.c:0:45 [opt]
    frame #30: 0x00000001000866b2 python.exe`_PyObject_FastCallDictTstate(tstate=0x0000000100486930, callable=0x0000000104053490, args=0x00007ffeefbfef70, nargsf=1, kwargs=0x00000001064c6d50) at call.c:152:15 [opt]
    frame #31: 0x00000001000875e4 python.exe`_PyObject_Call_Prepend(tstate=<unavailable>, callable=<unavailable>, obj=0x000000010622f3a0, args=<unavailable>, kwargs=<unavailable>) at call.c:482:24 [opt]
    frame #32: 0x00000001000ee1f9 python.exe`slot_tp_call(self=0x000000010622f3a0, args=0x000000010046c5a0, kwds=0x00000001064c6d50) at typeobject.c:7604:15 [opt]
    frame #33: 0x000000010008689c python.exe`_PyObject_MakeTpCall(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargs=<unavailable>, keywords=0x0000000104694960) at call.c:214:18 [opt]
    frame #34: 0x0000000100087049 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargsf=9223372036854775808, kwnames=0x0000000104694960) at pycore_call.h:90:16 [opt]
    frame #35: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #36: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f380, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #37: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #38: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102881b20, locals=0x0000000102830350, args=0x0000000000000000, argcount=0, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #39: 0x0000000100166615 python.exe`PyEval_EvalCode(co=<unavailable>, globals=0x0000000102830350, locals=0x0000000102830350) at ceval.c:1207:21 [opt]
    frame #40: 0x0000000100163b09 python.exe`builtin_exec_impl(module=<unavailable>, source=0x000000010617f230, globals=0x0000000102830350, locals=0x0000000102830350, closure=<unavailable>) at bltinmodule.c:1080:17 [opt]
    frame #41: 0x000000010016224d python.exe`builtin_exec(module=<unavailable>, args=0x0000000100c6f180, nargs=2, kwnames=0x0000000000000000) at bltinmodule.c.h:465:20 [opt]
    frame #42: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=<unavailable>, kwnames=0x0000000000000000) at methodobject.c:443:24 [opt]
    frame #43: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=9223372036854775810, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #44: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #45: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f0d8, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #46: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #47: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102c5eaf0, locals=0x0000000000000000, args=0x0000000102894ba8, argcount=2, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #48: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000102c5eaf0, stack=0x0000000102894ba8, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #49: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`_PyFunction_Vectorcall at call.c:385), callable=0x0000000102c5eaf0, tuple=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #50: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x0000000102c5eaf0, args=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #51: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #52: 0x00000001001d9f3d python.exe`pymain_run_module(modname=<unavailable>, set_argv0=<unavailable>) at main.c:300:14 [opt]
    frame #53: 0x00000001001d982d python.exe`pymain_run_python(exitcode=0x00007ffeefbff7dc) at main.c:595:21 [opt]
    frame #54: 0x00000001001d9718 python.exe`Py_RunMain at main.c:680:5 [opt]
    frame #55: 0x00000001001d99af python.exe`pymain_main(args=<unavailable>) at main.c:710:12 [opt]
    frame #56: 0x00000001001d99fb python.exe`Py_BytesMain(argc=<unavailable>, argv=<unavailable>) at main.c:734:12 [opt]
    frame #57: 0x0000000100001999 python.exe`main(argc=<unavailable>, argv=<unavailable>) at python.c:15:12 [opt]
    frame #58: 0x00007fff20417f3d libdyld.dylib`start + 1

Your environment

• CPython versions tested on: 3.11 current head
• Operating system and architecture: macOS

I put print statements in that show the file it crashes on is '/usr/local/lib/python3.11/site-packages/pytest_cov/plugin.py', but just running Python on that file doesn't reproduce the crash. I'll try to debug some more.

[JelleZijlstra]

Simple repro:

from ast import *
tree = Module(body=[
    Import(names=[alias(name='builtins', lineno=1, col_offset=0)], lineno=1, col_offset=0), 
    Import(names=[alias(name='traceback', lineno=0, col_offset=0)], lineno=0, col_offset=1)
], type_ignores=[])
compile(tree, "x", "exec", dont_inherit=True)

% ./python.exe wronglinenos.py 
Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.
zsh: abort      ./python.exe wronglinenos.py

[kumaraditya303]

944fffe is the first bad commit

commit 944fffee8916cb94321fa33cd3a43f4108717746
Author: Mark Shannon <mark@hotpy.org>
Date:   Thu Apr 21 16:10:37 2022 +0100

    GH-88116: Use a compact format to represent end line and column offsets. (GH-91666)
    
    * Stores all location info in linetable to conform to PEP 626.
    
    * Remove column table from code objects.
    
    * Remove end-line table from code objects.
    
    * Document new location table format

[iritkatriel]

@markshannon

Looks like the ast defaults the end_lineno to None, which then becomes 0 in the compiler. Maybe the ast should default it be equal to lineno? Or should None become -1 in c?

[markshannon]

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

[iritkatriel]

cc @pablogsal

[pablogsal]

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

That's what we do in some parts that deal with these information. I can prepare a PR

[pablogsal]

The question is if we should manage this in the AST creation routines or in the compiler. I assume that the first is more resilient but what do others think?

[The-Compiler]

FWIW, this seems to affect pytest's assertion rewriting as well. As soon as I'm running pytest against e.g.:

def test_foo():
    pass

with a Python 3.11.0b1 configured with --with-pydebug, I get:

collecting ... python: Python/compile.c:7531: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007f2555185740 (most recent call first):
  File ".../site-packages/_pytest/assertion/rewrite.py", line 361 in _rewrite_test
  File ".../site-packages/_pytest/assertion/rewrite.py", line 159 in exec_module
  ...

which is here:

https://github.com/pytest-dev/pytest/blob/c988e49af63a50f0704a46a93e73463666451af0/src/_pytest/assertion/rewrite.py#L361

and indeed i_end_lineno is 0:

(gdb) pp i
i = 
   autoderefcount="1",[
      i_opcode = <int> = {"100"}
      i_oparg = <int> = {"0"}
      i_target = <struct basicblock_*> = {"0x0"}
      i_except = <struct basicblock_*> = {"0x0"}
      i_lineno = <int> = {"1"}
      i_end_lineno = <int> = {"0"}
      i_col_offset = <int> = {"0"}
      i_end_col_offset = <int> = {"0"}
   ],<struct instr> = {"{...}"}

Turning pytest's assertion rewriting off with --assert=plain helps, and I can confirm it works fine before 944fffe.

Passing the end_lineno where we generate fake imports for the pytest assertion rewriting helpers seems to help:

diff --git i/src/_pytest/assertion/rewrite.py w/src/_pytest/assertion/rewrite.py
index 81096764e..5d7392dd4 100644
--- i/src/_pytest/assertion/rewrite.py
+++ w/src/_pytest/assertion/rewrite.py
@@ -727,7 +727,7 @@ def run(self, mod: ast.Module) -> None:
                 ast.alias("_pytest.assertion.rewrite", "@pytest_ar"),
             ]
         imports = [
-            ast.Import([alias], lineno=lineno, col_offset=0) for alias in aliases
+            ast.Import([alias], lineno=lineno, end_lineno=item.end_lineno, col_offset=0) for alias in aliases
         ]
         mod.body[pos:pos] = imports
 

cc @bluetech - though I suppose we don't necessarily need to fix this in pytest, hence I didn' t open an issue over there.

[The-Compiler]

Another project affected in the wild seems to be flask:

$ python3.11 -X faulthandler -c "import flask; flask.Flask('test')"
python: Python/compile.c:7534: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007fd44da08740 (most recent call first):
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1073 in _compile_builder
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 885 in compile
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 805 in bind
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1546 in add
  File ".../lib/python3.11/site-packages/flask/app.py", line 1086 in add_url_rule
  File ".../lib/python3.11/site-packages/flask/scaffold.py", line 56 in wrapper_func
  File ".../lib/python3.11/site-packages/flask/app.py", line 511 in __init__
  File "<string>", line 1 in <module>

[pablogsal]

I'm working on a patch, will have a draft PR ready soon