[Python] Interchange object data buffer has the wrong dtype / `from_dataframe` incorrect

[stinodego]

Describe the bug, including details regarding any error messages, version, and platform.

Code example

from datetime import datetime
import pyarrow as pa

string = pa.array(["a", "bc", None])
datetime = pa.array([datetime(2022, 1, 1), datetime(2022, 1, 2), datetime(2022, 1, 3)])
categorical = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0]), pa.array(["a", "b"]))

df = pa.Table.from_arrays(
    [string, datetime, categorical],
    names=["string", "datetime", "categorical"],
)

dfi = df.__dataframe__()

col = dfi.get_column_by_name("string")
print(col.dtype)
# (<DtypeKind.STRING: 21>, 8, 'u', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.STRING: 21>, 8, 'u', '=') -> SHOULD  BE: (<DtypeKind.UINT: 1>, 8, 'C', '=')

col = dfi.get_column_by_name("datetime")
print(col.dtype)
# (<DtypeKind.DATETIME: 22>, 64, 'tsu:', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.DATETIME: 22>, 64, 'tsu:', '=') -> SHOULD BE: (<DtypeKind.INT: 0>, 64, 'l', '=')

col = dfi.get_column_by_name("categorical")
print(col.dtype)
# (<DtypeKind.CATEGORICAL: 23>, 64, 'L', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.INT: 0>, 64, 'l', '=') -> CORRECT!

Issue description

As you can see, the dtype of the data buffer is the same as the dtype of the column. This is only correct for integers and floats. Categoricals, strings, and datetime types have a some integer as their physical representation. The data buffer should have this physical data type associated with it.

The dtype of the Column object should provide information on how to interpret the various buffers. The dtype associated with each buffer should be the dtype of the actual data in that buffer. This is the second part of the issue: the implementation of from_dataframe is incorrect - it should use the column dtype rather than the data buffer dtype.

Fix

Fixing the get_buffers implementation should be relatively simple. However, this will break any from_dataframe implementation (also from other libraries) that rely on the data buffer having the column dtype.

So fixing this should ideally go in three steps:

1. Fix the from_dataframe implementation to use the column dtype rather than the data buffer dtype to interpret the buffers.
2. Make sure other libraries have also updated their from_dataframe implementation. See BUG: Interchange object data buffer has the wrong dtype / from_dataframe incorrect pandas-dev/pandas#54781 for the pandas issue.
3. Fix the data buffer dtypes.

Tagging @AlenkaF as I know you've been working on the protocol for pyarrow.

Component(s)

Python

[AlenkaF]

Thank you for bringing this to our attention!
And thanks for putting effort to update the behaviour across all the libraries implementing the protocol.

One thought that comes to mind is that this should be made more explicit in the __dataframe__ protocol spec. Is there and issue opened as I haven't found one?

[stinodego]

One thought that comes to mind is that this should be made more explicit in the dataframe protocol spec. Is there and issue opened as I haven't found one?

There is no issue on this. I thought about opening one, but I would say that the text for the get_buffers method is quite clear already (see below). And from_dataframe is not explicitly part of the API, so there is no real documentation on how that should work.

Thanks so much for picking this up!

    def get_buffers(self) -> ColumnBuffers:
        """
        Return a dictionary containing the underlying buffers.

        The returned dictionary has the following contents:

            - "data": a two-element tuple whose first element is a buffer
                      containing the data and whose second element is the data
                      buffer's associated dtype.
            - "validity": a two-element tuple whose first element is a buffer
                          containing mask values indicating missing data and
                          whose second element is the mask value buffer's
                          associated dtype. None if the null representation is
                          not a bit or byte mask.
            - "offsets": a two-element tuple whose first element is a buffer
                         containing the offset values for variable-size binary
                         data (e.g., variable-length strings) and whose second
                         element is the offsets buffer's associated dtype. None
                         if the data buffer does not have an associated offsets
                         buffer.
        """

[pitrou]

There is no issue on this. I thought about opening one, but I would say that the text for the get_buffers method is quite clear already (see below).

If several implementations got this wrong, then the doc is certainly not clear enough IMHO.

[stinodego]

There is no issue on this. I thought about opening one, but I would say that the text for the get_buffers method is quite clear already (see below).

If several implementations got this wrong, then the doc is certainly not clear enough IMHO.

Not necessarily. I think everyone just followed along with the first implementation (pandas?) without thinking too hard about it.

But of course you could open a PR at the DataFrame API repo with an improved docstring.

[pitrou]

But of course you could open a PR at the DataFrame API repo with an improved docstring.

I'll try to. Though I already opened 5 issues and never got a single response...

[jorisvandenbossche]

And I think also several people involved in the protocol spec repo itself (me included) are clearly confusing this (apart from the implementations). Seeing data-apis/dataframe-api#87, where "I assume that the dtype should be provided as the same tuple returned from Column.dtype" was confirmed.
And the PR referenced there to add the protocol to StaticFrame did exactly the same "mistake".

[jorisvandenbossche]

So yes, I think although strictly speaking the current docstring is clear and unambiguous, calling it out explicitly would clearly be worth it.