[Python] RecordBatchReader.cast() does not work when reading CSV file.

[pstrzelczak]

Describe the bug, including details regarding any error messages, version, and platform.

While investigating pyarrow's capability to read large csv files in streaming fashion I created a simple csv file with a single mixed_column column and 1M rows with value 1 and a single last row with value A. After I found out type inference happens for the first block only I noticed RecordBatchReader.cast() method. While using it does not satisfy my needs because that is being able to infer data types for all record batches separately, I experimented with this cast() method and found it seemed not to work.

Reproduction:

import pyarrow.csv 
import pyarrow.lib
import pyarrow

def cast_test():
    with pyarrow.csv.open_csv('/tmp/mixed.csv') as r:
        schema = r.schema
        updated_schema = schema.set(0, pyarrow.field("mixed_column", pyarrow.string()))
        r_cast = r.cast(updated_schema)
        for batch in r_cast:
            print(batch)
cast_test()

Ouptut:

$ python scripts/cast_test.py 
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
Traceback (most recent call last):
  File "/home/pstrzelczak/scripts/cast_test.py", line 12, in <module>
    cast_test()
  File "/home/pstrzelczak/scripts/cast_test.py", line 10, in cast_test
    for batch in r_cast:
  File "pyarrow/ipc.pxi", line 671, in pyarrow.lib.RecordBatchReader.__next__
  File "pyarrow/ipc.pxi", line 705, in pyarrow.lib.RecordBatchReader.read_next_batch
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: In CSV column #0: CSV conversion error to int64: invalid value 'A'

I am using pyarrow 19.0.1 on Ubuntu 24.04 with Python 3.10.

I found this functionality was implemented in #40070 by @paleolimbot

Component(s)

Python

[davlee1972]

That's because the first batch is all 1s so the initial schema it generates for the recordbatchreader is Int64 for the column.

The first batch is successfully read in as Int64s and then casted from Int64 to mixed_column: string

The second batch fails to read "A" in it as a Int64 which happens before cast().

The following works..
Explicitly map column names to column types. Passing this argument disables type inference on the defined columns.

with open("/tmp/mixed.csv", "w") as f:
    f.write("mixed_column\n")
    f.write("1\n" * 1000000)
    f.write("A\n")

import pyarrow.csv 

def cast_test():
    c_options = pyarrow.csv.ConvertOptions(column_types={"mixed_column": "string"})
    print(c_options)
    with pyarrow.csv.open_csv('/tmp/mixed.csv', convert_options=c_options) as r:
        for batch in r:
            print(batch)

cast_test()

<pyarrow._csv.ConvertOptions object at 0x7f6c85809780>
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","A"]

Personally I use pyarrow.dataset for everything instead of the csv, parquet, etc. classes.

[pstrzelczak]

The second batch fails to read "A" in it as a Int64 which happens before cast().

I thought the semantics of RecordBatchReader.cast() is that raw stream will no longer be interpreted according to inferred schema but target_schema (argument of cast).