Update Parquet.net to latest version

[zweistein22]

Microsoft.ML.Parquet must use Parquet.net version == 2.1.3 . It may not use the current Parquet.net versions 3.xx (3.91) as the internal structures have changed and Parquet.Data.Dataset does not exist in recent Parquet.net versions.

When a newer Parquet.dll is found however this newer version is used and causes a crash al Assembly.Load

https://github.com/Rokin-Tech/machinelearning/tree/main/src/Microsoft.ML.Parquet
Best solution would be to update Microsoft.ML.Parquet .

second best is to force linkage of exact parquet.net 2.1.3 and not allow >= 2.1.3

Best regards
Andreas

[zweistein22]

tentative implementation:

https://github.com/Rokin-Tech/machinelearning/blob/d2fd481d8cac78fb6f941d6173b11d9a3082557d/src/Microsoft.ML.Parquet/ParquetLoader.cs

[michaelgsharp]

It may be easier to just lock the version until we are able to fully get it updated. Does your tentative implementation fully work?

[zweistein22]

My version works for simpler parquet Schemas. Byte[] Columns work, but string[] columns not. But I assume this did not work in the original implementation either.
All value types work.
I had difficulties locking to the 2.3.1 version as I have to use it it side by side in the process with 3.9.1 , was easier to write this tentative implementation.

[zweistein22]

Modified version: Seems to work properly as long as the row count is less than the predefined ColumnChunkReadSize , by default set to 1 Million.

So to read parquet files with more than 1 Mill rows one has to set the ColumnChunkReadSize to be more than the total rows.
Seems that this error/behaviour is same as in the original file.

Temporary fix:

var parquetAssembly = Assembly.Load("Microsoft.ML.Parquet");
var parquetLoarderType = parquetAssembly.GetType("Microsoft.ML.Data.ParquetLoader");
var parquetArgsType = parquetLoarderType.GetNestedType("Arguments");
var argc = parquetArgsType.GetConstructors()[0];
var args = argc.Invoke(null);
Type t = args.GetType();
FieldInfo pi = t.GetField("ColumnChunkReadSize");
pi.SetValue(args, 10000000);
var methods = parquetLoarderType.GetConstructor(new[] { typeof(IHostEnvironment), parquetArgsType, typeof(string) });
var p = methods.Invoke(new object[] { ctx, args, path });

The problem shows up only when there are Array type columns like float[] . It does not show up with byte[] Columns,

[michaelgsharp]

Thats good to know. Thanks for that info.

I'm going to update your issue to a feature request to update to use the latest version of the Parquet packages. It would be a good thing to do.

[michaelgsharp]

@luisquintanilla for prioritization since its now a feature request.