Error creating table from pyarrow schema with pa.uuid()

[simw]

Apache Iceberg version

0.9.0 (latest release)

Please describe the bug 🐞

Preamble: using a local sqlite db:

from pyiceberg.catalog import load_catalog

warehouse_path = "data/warehouse"
catalog = load_catalog(
    "default",
    **{
        'type': 'sql',
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)

A pyiceberg UUID column works fine:

from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, UUIDType

schema = Schema(
    NestedField(field_id=1, name="uuid", field_type=UUIDType(), required=False),
)

catalog.create_table("default.test2", schema=schema)

But a pyarrow UUID column gives an error:

import pyarrow as pa

schema = pa.schema([pa.field("foo", pa.uuid(), nullable=True)])

catalog.create_table("default.test4", schema=schema)

The exception is:

File ~/Code/Projects/others/icebergs/.venv/lib/python3.13/site-packages/pyiceberg/io/pyarrow.py:1032, in _(obj, visitor)
   1030     result = visit_pyarrow(field_type, visitor)
   1031 except TypeError as e:
-> 1032     raise UnsupportedPyArrowTypeException(obj, f"Column '{obj.name}' has an unsupported type: {field_type}") from e
   1033 visitor.after_field(obj)
   1035 return visitor.field(obj, result)

UnsupportedPyArrowTypeException: Column 'foo' has an unsupported type: extension<arrow.uuid>

Related to simw/pydantic-to-pyarrow#27

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[jim-ngoo]

we have the UUIDType type already, I think what we missed is the visit_pyarrow decorator

I think we need to handle the case pa.uuid() here:

iceberg-python/pyiceberg/io/pyarrow.py

Line 1203 in 260ef54

def primitive(self, primitive: pa.DataType) -> PrimitiveType:

[dingo4dev]

iceberg-python/pyiceberg/io/pyarrow.py

Lines 587 to 592 in 8bfb16c

	def schema_to_pyarrow(
	schema: Union[Schema, IcebergType],
	metadata: Dict[bytes, bytes] = EMPTY_DICT,
	include_field_ids: bool = True,
	) -> pa.schema:
	return visit(schema, _ConvertToArrowSchema(metadata, include_field_ids))

The UUIDType transfer to pyarrow is fixed_size_binary[16] . You might use pa.binary(16) and store the bytes of the uuid in your pyarrow table.

import uuid
import pyarrow as pa
uuids = pa.array([uuid.uuid4().bytes for _ in range(100) ], pa.binary(16))

[Tishj]

I think I've also encountered this problem, trying to write tests for https://github.com/duckdb/duckdb-iceberg
I am using a build from main (0.10 dev)

The table.append(...) method won't accept this array:

col_uuid = pa.array([UUID('020d4fc7-acd6-45ac-b216-7873f4038e1f').bytes], pa.uuid())

Results in:

  File "/iceberg-python/pyiceberg/io/pyarrow.py", line 1061, in _
    raise UnsupportedPyArrowTypeException(obj, f"Column '{obj.name}' has an unsupported type: {field_type}") from e
pyiceberg.io.pyarrow.UnsupportedPyArrowTypeException: Column 'col_uuid' has an unsupported type: extension<arrow.uuid>

What does work is this:

col_uuid = pa.array([UUID('020d4fc7-acd6-45ac-b216-7873f4038e1f').bytes], pa.binary(16))

But the problem then is that the parquet file created by pyiceberg does not have the UUIDType() logical type for the field, which trips up our (duckdb's) parquet reader.

I think it's fine to accept pa.binary(16), but then pyiceberg should add the UUIDType() logical type to the parquet file's field, if the destination is a uuidtype, such as this:

update.add_column("col_uuid", UUIDType(), default_value="f79c3e09-677c-4bbd-a479-3f349cb785e7", required=False)

[simw]

This would be fixed by #2007

[Fokko]

@simw Indeed, I think the PR is ready now 👍

[Tishj]

I just tested, I don't think this is fixed?
I've build from source, creating version 0.10.0

I still can't use pa.uuid() to populate the table
I can use pa.binary(16) and that still doesn't add the UUIDType logical type to the parquet files

Relevant code from the PR:

    arr_table = pa.Table.from_pydict(
        {
            "uuid": [
                uuid.UUID("00000000-0000-0000-0000-000000000000").bytes,
                uuid.UUID("11111111-1111-1111-1111-111111111111").bytes,
            ],
        },
        schema=pa.schema(
            [
                # Uuid not yet supported, so we have to stick with `binary(16)`
                # https://github.com/apache/arrow/issues/46468
                pa.field("uuid", pa.binary(16), nullable=False),
            ]
        ),
    )

From what I gather the issue is pyarrow, which is fine, but I don't understand closing this issue if that's the case

[kevinjqliu]

Good catch, i think this was auto-closed when the PR was merged

I was able to verify that this issue still exists by changing pa.binary(16) to pa.uuid() in test_uuid_partitioning

poetry run pytest tests/integration/test_writes/test_writes.py::test_uuid_partitioning -m integration

As Fokko mentioned in #2007, this is dependent on newer pyarrow version, likely 21.0.0 according to apache/arrow#46469

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'