[Python] Illegal hardware instruction on pyarrow import

[asfimport]

I can't import pyarrow, it crashes:

>>> import pyarrow as pa
[1]    31441 illegal hardware instruction (core dumped)  python3

Core dump is attached to issue, it can help you to understand what is the problem.

The environment is:

Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas: 0.23.4
NumPy: 1.15.4
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Environment: Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas: 0.23.4
NumPy: 1.15.4
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Elchin

Original Issue Attachments:

• core

_{Note: This issue was originally created as ARROW-4272. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Without the exact same binaries and libraries as you, it will be difficult to do something with the core dump.
Could you run gdb python3.6 core ? Then under the gdb prompt type bt to get the C backtrace and post the result here.

[asfimport]

Elchin:
@pitrou of course:

#0  0x00007f445ba46070 in std::pair<std::__detail::_Node_iterator<std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> >, false, true>, bool> std::_Hashtable<absl::string_view, std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> >, std::allocator<std::pair<absl::string_view const, std::function<bool (tensorflow::Variant*)> > >, std::__detail::_Select1st, std::equal_to<absl::string_view>, tensorflow::hash<absl::string_view, void>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::pair<absl::string_view, std::function<bool (tensorflow::Variant*)> > >(std::integral_constant<bool, true>, std::pair<absl::string_view, std::function<bool (tensorflow::Variant*)> >&&) () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#1  0x00007f445ba47f25 in tensorflow::UnaryVariantOpRegistry::RegisterDecodeFn(std::string const&, std::function<bool (tensorflow::Variant*)> const&) ()
   from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#2  0x00007f445ba2467c in tensorflow::variant_op_registry_fn_registration::UnaryVariantDecodeRegistration<tensorflow::Tensor>::UnaryVariantDecodeRegistration(std::string const&) () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#3  0x00007f445b991885 in _GLOBAL__sub_I_tensor.cc () from /home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f446e520733 in call_init (env=0xc109f0, argv=0x7ffc89f96088, argc=1, l=<optimized out>) at dl-init.c:72
#5  _dl_init (main_map=main_map@entry=0x14a1090, argc=1, argv=0x7ffc89f96088, env=0xc109f0) at dl-init.c:119
#6  0x00007f446e5251ff in dl_open_worker (a=a@entry=0x7ffc89f941b0) at dl-open.c:522
#7  0x00007f446e2862df in __GI__dl_catch_exception (exception=exception@entry=0x7ffc89f94190, operate=operate@entry=0x7f446e524dc0 <dl_open_worker>,
    args=args@entry=0x7ffc89f941b0) at dl-error-skeleton.c:196
#8  0x00007f446e5247ca in _dl_open (file=0x7f445c6ed8c0 "/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so", mode=-2147483646,
    caller_dlopen=0x7f4466aa4cda <py_dl_open+90>, nsid=<optimized out>, argc=1, argv=<optimized out>, env=0xc109f0) at dl-open.c:605
#9  0x00007f446dcfcf96 in dlopen_doit (a=a@entry=0x7ffc89f943e0) at dlopen.c:66
#10 0x00007f446e2862df in __GI__dl_catch_exception (exception=exception@entry=0x7ffc89f94380, operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>,
    args=args@entry=0x7ffc89f943e0) at dl-error-skeleton.c:196
#11 0x00007f446e28636f in __GI__dl_catch_error (objname=objname@entry=0xafbf20, errstring=errstring@entry=0xafbf28, mallocedp=mallocedp@entry=0xafbf18,
    operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>, args=args@entry=0x7ffc89f943e0) at dl-error-skeleton.c:215
#12 0x00007f446dcfd735 in _dlerror_run (operate=operate@entry=0x7f446dcfcf40 <dlopen_doit>, args=args@entry=0x7ffc89f943e0) at dlerror.c:162
#13 0x00007f446dcfd051 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#14 0x00007f4466aa4cda in py_dl_open () at ./Modules/_ctypes/callproc.c:1348
#15 0x0000000000502d6f in _PyCFunction_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=<optimized out>,
    func_obj=<built-in method dlopen of module object at remote 0x7f4466ceb188>) at ../Objects/methodobject.c:234
#16 _PyCFunction_FastCallKeywords (kwnames=<optimized out>, nargs=<optimized out>, stack=<optimized out>, func=<optimized out>) at ../Objects/methodobject.c:294
#17 call_function.lto_priv () at ../Python/ceval.c:4837
#18 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#19 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446c756448, for file /usr/lib/python3.6/ctypes/__init__.py, line 348, in __init__ (name='/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so', mode=0, handle=None, use_errno=False, use_last_error=False, _FuncPtr=<_ctypes.PyCFuncPtrType at remote 0x14a0cf8>)) at ../Python/ceval.c:4166
#20 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#21 0x0000000000501ba7 in _PyFunction_FastCallDict () at ../Python/ceval.c:5070
#22 0x0000000000591461 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffc89f947c0, func=<function at remote 0x7f4466ceabf8>) at ../Objects/abstract.c:2310
#23 _PyObject_Call_Prepend (kwargs=0x0, args=<optimized out>, obj=<optimized out>, func=<function at remote 0x7f4466ceabf8>) at ../Objects/abstract.c:2373
#24 method_call.lto_priv () at ../Objects/classobject.c:317
#25 0x000000000054b813 in PyObject_Call (kwargs=0x0, args=('/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so',),
    func=<method at remote 0x7f446e61fb88>) at ../Objects/abstract.c:2261
#26 slot_tp_init () at ../Objects/typeobject.c:6420
#27 0x0000000000555421 in type_call.lto_priv () at ../Objects/typeobject.c:915
#28 0x00000000005a730c in _PyObject_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<type at remote 0xc56148>)
    at ../Objects/tupleobject.c:131
#29 _PyObject_FastCallKeywords () at ../Objects/abstract.c:2496
#30 0x0000000000503073 in call_function.lto_priv () at ../Python/ceval.c:4861
#31 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#32 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x14c2858, for file /home/elch10/.local/lib/python3.6/site-packages/pyarrow/compat.py, line 229, in import_tensorflow_extension (os=<module at remote 0x7f446e5ce4a8>, tensorflow_loaded=False, path='/home/elch10/.local/lib/python3.6/site-packages/tensorflow', ext='/home/elch10/.local/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so', ctypes=<module at remote 0x7f44690d9548>)) at ../Python/ceval.c:754
#33 _PyFunction_FastCall (globals=<optimized out>, nargs=21768280, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#34 fast_function.lto_priv () at ../Python/ceval.c:4954
#35 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#36 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#37 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb77f08, for file /home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py, line 50, in <module> ()) at ../Python/ceval.c:4166
#38 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#39 0x0000000000511d78 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0,
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), _co=<code at remote 0x7f446ce3fc90>) at ../Python/ceval.c:4187
#40 PyEval_EvalCode (
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), co=<code at remote 0x7f446ce3fc90>) at ../Python/ceval.c:731
#41 builtin_exec_impl (module=<optimized out>,
    locals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated),
    globals={'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '_...(truncated), source=<code at remote 0x7f446ce3fc90>) at ../Python/bltinmodule.c:983
#42 builtin_exec.lto_priv () at ../Python/clinic/bltinmodule.c.h:283
#43 0x000000000056617e in PyCFunction_Call () at ../Objects/methodobject.c:112
#44 0x000000000050bb66 in do_call_core (kwdict={},
    callargs=(<code at remote 0x7f446ce3fc90>, {'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsi...(truncated), func=<built-in method exec of module object at remote 0x7f446e704638>) at ../Python/ceval.c:5102
#45 _PyEval_EvalFrameDefault () at ../Python/ceval.c:3404
#46 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446e588930, for file <frozen importlib._bootstrap>, line 219, in _call_with_frames_removed (f=<built-in method exec of module object at remote 0x7f446e704638>, args=(<code at remote 0x7f446ce3fc90>, {'__name__': 'pyarrow', '__doc__': None, '__package__': 'pyarrow', '__loader__': <SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, '__spec__': <ModuleSpec(name='pyarrow', loader=<...>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, '__path__': [...], '__file__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', '__cached__': '/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__i...(truncated)) at ../Python/ceval.c:4166
#47 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#48 0x0000000000502540 in fast_function.lto_priv () at ../Python/ceval.c:4978
#49 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#50 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#51 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446e5dfcf8, for file <frozen importlib._bootstrap_external>, line 678, in exec_module (self=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, module=<module at remote 0x7f446ce5b318>, code=<code at remote 0x7f446ce3fc90>))
    at ../Python/ceval.c:754
#52 _PyFunction_FastCall (globals=<optimized out>, nargs=139931886157048, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#53 fast_function.lto_priv () at ../Python/ceval.c:4954
#54 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#55 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#56 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446ced2448, for file <frozen importlib._bootstrap>, line 665, in _load_unlocked (spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/
#56 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7f446ced2448, for file <frozen importlib._bootstrap>, line 665, in _load_unlocked (spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/---Type <return> to continue, or q <return> to quit---
__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>, module=<module at remote 0x7f446ce5b318>))
    at ../Python/ceval.c:754
#57 _PyFunction_FastCall (globals=<optimized out>, nargs=139931861984328, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#58 fast_function.lto_priv () at ../Python/ceval.c:4954
#59 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#60 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#61 0x0000000000502209 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb78328, for file <frozen importlib._bootstrap>, line 955, in _find_and_load_unlocked (name='pyarrow', import_=<built-in method __import__ of module object at remote 0x7f446e704638>, path=None, parent='', spec=<ModuleSpec(name='pyarrow', loader=<SourceFileLoader(name='pyarrow', path='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py') at remote 0x7f446ce5da90>, origin='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__init__.py', loader_state=None, submodule_search_locations=['/home/elch10/.local/lib/python3.6/site-packages/pyarrow'], _set_fileattr=True, _cached='/home/elch10/.local/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', _initializing=True) at remote 0x7f446ce5dac8>)) at ../Python/ceval.c:754
#62 _PyFunction_FastCall (globals=<optimized out>, nargs=12026664, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#63 fast_function.lto_priv () at ../Python/ceval.c:4954
#64 0x0000000000502f3d in call_function.lto_priv () at ../Python/ceval.c:4858
#65 0x0000000000506859 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#66 0x0000000000501945 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xb1edb8, for file <frozen importlib._bootstrap>, line 971, in _find_and_load (name='pyarrow', import_=<built-in method __import__ of module object at remote 0x7f446e704638>, module=<object at remote 0x7f446e6c6030>)) at ../Python/ceval.c:754
#67 _PyFunction_FastCall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Python/ceval.c:4919
#68 _PyFunction_FastCallDict () at ../Python/ceval.c:5021
#69 0x00000000005a36f1 in _PyObject_FastCallDict () at ../Objects/abstract.c:2310
#70 0x000000000059662e in _PyObject_CallMethodIdObjArgs () at ../Objects/abstract.c:2796
#71 0x00000000004ee84d in PyImport_ImportModuleLevelObject () at ../Python/import.c:1578
#72 0x000000000050896c in import_name (level=0, fromlist=None, name='pyarrow', f=Frame 0x7f446e5df788, for file <stdin>, line 1, in <module> ())
    at ../Python/ceval.c:5231
#73 _PyEval_EvalFrameDefault () at ../Python/ceval.c:2899
#74 0x0000000000504c28 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f446e5df788, for file <stdin>, line 1, in <module> ()) at ../Python/ceval.c:4166
#75 _PyEval_EvalCodeWithName.lto_priv.1761 () at ../Python/ceval.c:4166
#76 0x0000000000506393 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>,
    globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4187
#77 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:731
#78 0x0000000000634d52 in run_mod () at ../Python/pythonrun.c:1025
#79 0x00000000004a38c5 in PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7f446e50aa00 <_IO_2_1_stdin_>, filename=filename@entry='<stdin>',
    flags=flags@entry=0x7ffc89f95e8c) at ../Python/pythonrun.c:246
#80 0x00000000004a5cd5 in PyRun_InteractiveLoopFlags (fp=fp@entry=0x7f446e50aa00 <_IO_2_1_stdin_>, filename_str=filename_str@entry=0x7541e0 "<stdin>",
    flags=flags@entry=0x7ffc89f95e8c) at ../Python/pythonrun.c:114
#81 0x00000000006387b3 in PyRun_AnyFileExFlags () at ../Python/pythonrun.c:75
#82 0x000000000063915a in run_file (p_cf=0x7ffc89f95e8c, filename=<optimized out>, fp=<optimized out>) at ../Modules/main.c:340
#83 Py_Main () at ../Modules/main.c:810
#84 0x00000000004a6f10 in main (argc=1, argv=0x7ffc89f96088) at ../Programs/python.c:69

[asfimport]

Antoine Pitrou / @pitrou:
Ah, you're using Tensorflow. I take it you installed both Tensorflow and PyArrow using pip?

Tensorflow publishes incompatible Python packages (wheels) that can make other libraries such as Arrow crash (#). I'm not 100% sure it's the problem here but that is quite likely.

Can you try using conda instead and install both libraries using conda?

Also, I think it would be helpful if you reported this problem to the Tensorflow project, due to the issue I mentioned above.

cc @pcmoritz

(#) More explanation here: https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/TMqRaT-H2bI

[asfimport]

Elchin:
@pitrou of course, with conda it works fine.

[abcbarryn]

I have this same issue and I do not have tensorflow installed.
Steps to reproduce:

pip install pyarrow

$ python
import pyarrow
illegal instruction

CPU and system information:

vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping	: 6
microcode	: 0xd2
cpu MHz		: 3000.116
cache size	: 4096 KB

Linux static1 3.16.7-48-desktop #1 SMP PREEMPT Wed Oct 26 18:09:22 UTC 2016 (8c027f6) x86_64 GNU/Linux

glibc-2.18-4.44.1.x86_64

OpenSuSE linux

[kou]

Could you open a new issue for your case with your Python version and full log of pip install pyarrow?

[abcbarryn]

Python versions tried: 3.9.4, 3.9.16, 3.9.13, 3.11
pip install pyarrow doesn't throw any errors, how do I view the log? It builds and install cleanly, but when I try to use it I get illegal instruction. I will open a new case shortly.

[lahwaacz]

It appears that pyarrow requires a CPU with SSE 4.1 due to the PINSRQ instruction:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install pyarrow
Collecting pyarrow
  Using cached pyarrow-16.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting numpy>=1.16.6 (from pyarrow)
  Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Using cached pyarrow-16.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.8 MB)
Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.0 MB)
Installing collected packages: numpy, pyarrow
Successfully installed numpy-1.26.4 pyarrow-16.0.0
$ objdump -S .venv/lib/python3.12/site-packages/pyarrow/lib.cpython-312-x86_64-linux-gnu.so | grep pinsrq
   6e8b2:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e8cd:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e8eb:	66 48 0f 3a 22 c2 01 	pinsrq $0x1,%rdx,%xmm0
   6e90d:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e93a:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e9df:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6e9fd:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
   6ea1b:	66 48 0f 3a 22 c0 01 	pinsrq $0x1,%rax,%xmm0
...

The first PINSRQ instruction gets called right at the module import.

[HanabishiRecca]

https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_USER_SIMD_LEVEL

by default, SSE4.2 optimizations are enabled on x86 builds: therefore, with this default setting, Arrow C++ does not work at all on a CPU without support for SSE4.2. This setting can be changed using the ARROW_SIMD_LEVEL CMake variable so as to either raise or lower the optimization level.