Consider backport (or not) FileSystemEnumerator fixes for Unix

[jozkee]

Opening this issue to track if we should backport our fixes in FileSystemEnumerator, our options are:

1. No(ish) risk fix - Increase buffer size for readir_r and add support for enumerating filenames > 255 bytes #116619, in summary, it increases the buffer size to the largest known today (1024 instead of 256). It isn't as future proof as Remove readdir_r entirely and handle filenames > 255 bytes #116639 because it still depends on a buffer size but opposite to 116639, it doesn't pose the non-zero risk of moving from one syscall to another which was defined as non-thread-safe for many years by POSIX.
2. Non-zero risk fix - Remove readdir_r entirely and handle filenames > 255 bytes #116639, while we are highly confident that no readdir implementation may pose a problem today, our obsession with compatibility still makes us hesitant to discard the risk. If we choose to backport this, it would be better to do it after preview6 so the change gets ingested by most users (scream test).
3. Do nothing - These fixes can be labeled as reliability, I don't think we can choose this path as there's user reports showing this is hittable and hard to diagnose. Both options above also contain the IndexOf<byte>(0) fix mentioned by @jkotas which in itself is worth backporting.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[jkotas]

there's user reports showing this is hittable and #107503 (comment).

Backporting the IndexOf(0) fix - that should address this user-reported issue - sounds reasonable to me.

I am not sure about the rest.

[jozkee]

@jkotas, we still need to either increase the buffer size or switch to readdir which does not need a user buffer to fix buffer overflows caused by readdir_r's memcpy.

[jkotas]

to fix buffer overflows caused by readdir_r's memcpy.

Is this musl behavior "by design" or a bug in musl? I have looked at this as a bug in musl.

[hamarb123]

Just mentioning this comment here also in case no one looks at the PR again: #116639 (comment)

[jozkee]

@jkotas, I agree this is most likely a bug in MUSL, though I haven’t seen an official statement yet. Increasing the buffer size of the pooled buffer will put us at a better state at relative no risk, but if that creates bad precedent, then I'm fine with not including it.

[jeffhandley]

I defer to you two, @jozkee and @jkotas, and support whichever approach you settle on for the backports to .NET 8 and 9.

[dje-dev]

The chess engine Ceres was competing at the TCEC competition (on .NET 8) and crashed with a stack trace suggesting this defect as the cause (stack trace below, reaches NonPackedIndexOfValueType via FileSystemEnumerator).

893805583 <Ceres 1.07_C1-640-34(477): 168 98468841578 STDERR Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
893805583 <Ceres 1.07_C1-640-34(477): 169 98468873968 STDERR    at System.SpanHelpers.NonPackedIndexOfValueType[[System.Byte, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.SpanHelpers+DontNegate`1[[System.Byte, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Byte ByRef, Byte, Int32)
893805583 <Ceres 1.07_C1-640-34(477): 170 98468877378 STDERR    at Interop+Sys+DirectoryEntry.GetName(System.Span`1<Char>)
893805583 <Ceres 1.07_C1-640-34(477): 171 98468879558 STDERR    at System.IO.Enumeration.FileSystemEntry.get_FileName()
893805583 <Ceres 1.07_C1-640-34(477): 172 98468881008 STDERR    at System.IO.Enumeration.FileSystemEntry.get_FullPath()
893805583 <Ceres 1.07_C1-640-34(477): 173 98468882578 STDERR    at System.IO.Enumeration.FileSystemEntry.Initialize(System.IO.Enumeration.FileSystemEntry ByRef, DirectoryEntry, System.ReadOnlySpan`1<Char>, System.ReadOnlySpan`1<Char>, System.ReadOnlySpan`1<Char>, System.Span`1<Char>)
893805583 <Ceres 1.07_C1-640-34(477): 174 98468884888 STDERR    at System.IO.Enumeration.FileSystemEnumerator`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
893805583 <Ceres 1.07_C1-640-34(477): 175 98468886408 STDERR    at System.Collections.Generic.List`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)
893805583 <Ceres 1.07_C1-640-34(477): 176 98468890117 STDERR    at System.IO.Directory.GetFiles(System.String, System.String, System.IO.EnumerationOptions)

[jozkee]

I don't think we need to service this, we can instead recommend migrating to .NET 10.