[Python][Packaging] Support Python 3.12 and upload wheels to PyPI

[EwoutH]

Describe the enhancement requested

Currently there are no pyarrow wheels for Python 3.12 on PyPI. Python 3.12 releases in a week (Monday 2023-10-02), so it would be great if wheels could be build and uploaded to PyPI, especially since many packages depend on it.

Component(s)

Python

[AlenkaF]

@raulcd

[raulcd]

Hi @EwoutH , thanks for opening the issue. Due to the pyarrow wheels being currently an artifact of the release for Apache Arrow we can't release them independently and we have to do an Arrow release. We have a scheduled release (14.0.0) in the next weeks. Code freeze is expected for the 10th of October. Due to the current complexity on the release process and the Apache guidelines unfortunately they won't be available just yet but I'll work on creating the pyarrow wheels for Python 3.12 and will add it to the 14.0.0 release.
As soon as they are available if necessary you'll be able to test with our nightly development wheels, those are not official releases but can help development and testing: https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages
cc ~ @jorisvandenbossche

[jsherman256]

FYI: this impacts docker images based on python:3 if they try to install pyarrow or packages dependent on pyarrow. This is because the python:3 image now refers to python 3.12, but pyarrow cannot be built inside that image by default, since it's missing cmake:

#7 381.6   Building wheel for pyarrow (pyproject.toml): started
#7 382.1   Building wheel for pyarrow (pyproject.toml): finished with status 'error'
#7 382.1   error: subprocess-exited-with-error
#7 382.1   
#7 382.1   × Building wheel for pyarrow (pyproject.toml) did not run successfully.
...
#7 382.1       error: command 'cmake' failed: No such file or directory
#7 382.1       [end of output]
#7 382.1   
#7 382.1   note: This error originates from a subprocess, and is likely not a problem with pip.
#7 382.1   ERROR: Failed building wheel for pyarrow

Just leaving this here in case it helps someone else debugging why their python container suddenly wont build anymore. I worked around this by moving from python:3 to python:3.11.6