[Python] Incorrect conversion from datetime.datetime and datetime.time objects with non-UTC timezones

[Wainberg]

Describe the bug, including details regarding any error messages, version, and platform.

I may be missing something fundamental, but Pyarrow seems to improperly handle the conversion from datetime.datetime and datetime.time objects with non-UTC timezones.

Datetimes

This Python datetime is the turn of the millenium in UTC+1:

>>> dt = datetime.datetime(2000, 1, 1, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))

Converting to pyarrow makes it 11 PM in UTC+1, which is a different time (not just the same time in a different coordinate system):

>>> a = pa.array([dt])
>>> a
<pyarrow.lib.TimestampArray object at 0x7f4d0197e3e0>
[
  1999-12-31 23:00:00.000000
]
>>> a.type
TimestampType(timestamp[us, tz=+01:00])

When converted to R (using the Arrow C API), this becomes the following POSIXct vector:

> a
[1] "1999-12-31 23:00:00"
> attr(a, 'tzone')
[1] "+01:00"

which is again 11 PM in UTC+1.

So the Arrow -> R conversion is consistent, but the Python -> Arrow conversion seems like a bug.

Times

Creating a pyarrow array from a datetime.time strips timezone info:

>>> pa.array([datetime.time(12, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))]) == \
... pa.array([datetime.time(12, 0)])
True

which seems to be intentional according to https://arrow.apache.org/docs/python/generated/pyarrow.Time64Array.html:

Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.

But again, these are two different times (not just the same time in a different coordinate system), so they shouldn't compare equal.

This is on pyarrow 14.0.2, Python 3.12, R version 4.3.2, R Arrow version 14.0.1.

Component(s)

Python

[jorisvandenbossche]

I think everything is behaving as expected, but the repr is confusing.

Datetimes
...
Converting to pyarrow makes it 11 PM in UTC+1, which is a different time (not just the same time in a different coordinate system):

So it actually did convert it to 11 PM in UTC (the UTC value is what is stored under the hood), and thus still representing midnight in UTC+1.
One might expect wall-time relative to the type in the repr, but so the repr is showing the UTC values, which is causing the confusion.

In the latest pyarrow 15.0, we improved the repr a little bit by adding "Z" as an indicator that what you see are UTC values:

In [4]: a
Out[4]: 
<pyarrow.lib.TimestampArray object at 0x7f2fe814ebc0>
[
  1999-12-31 23:00:00.000000Z
]

In [5]: a.type
Out[5]: TimestampType(timestamp[us, tz=+01:00])

Times

Creating a pyarrow array from a datetime.time strips timezone info:

The Time type in Arrow format does not have a notion of timezones (see fatbuffer spec), and pyarrow's conversion code for python->arrow currently just ignores the timezone.
This is not ideal, and maybe it would be better if pyarrow just raised an error instead? (indicating it doesn't support datetime.time objects with a timezone?)

which seems to be intentional according to https://arrow.apache.org/docs/python/generated/pyarrow.Time64Array.html:

Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.

Hmm, that note seems quite outdated (also, it is about timestamps, not about times .. although it appears in the Time64Array doc page, but that's because this from_pandas method is implemented on the base class and thus has the same docstring for all concrete Array subclasses).

[jorisvandenbossche]

Hmm, that note seems quite outdated

That note seems to have been added almost at the beginning of the project (#287), and since then quite some changes were made to timezone handling.
(localized timestamps are not exactly returned as UTC, but the timezone information is preserved; and for timezone-naive data there is an equivalent timestamp without timezone type in Arrow ..)

[Wainberg]

Makes sense! You're right that the datetime.datetimes are all the same between base Python, pandas, Arrow and base R, they're just displayed differently. I definitely prefer base Python and pandas's display to Arrow's though, maybe version 16 could switch? ;)

>>> pa.array([datetime.datetime(2000, 1, 1, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))])
<pyarrow.lib.TimestampArray object at 0x7fa7b9084280>
[
  1999-12-31 23:00:00.000000
]
>>> pa.array([datetime.datetime(2000, 1, 1, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))]).to_pandas()
0   2000-01-01 00:00:00+01:00
dtype: datetime64[us, pytz.FixedOffset(60)]

For the times, I would definitely raise an error on datetime.time objects with a non-default timezone, it seems quite dangerous to truncate the timezone silently.