Emscripten metadata file size is wrong

[juntyr]

I tried this code:

std::fs::read("my-file")

for a file that is <1MB large.

I expected to see this happen: The read should allocate a vec that is as large as the file content. The heap size in Emscripten should not meaningfully change.

Instead, this happened: The Emscripten heap grows by 1.8GB

I looked into the implementation of fs::read. Since it uses the file metadata size to pre-allocate the output vector, I debug printed the file size, which is around 1.8GB.

Since my Rust code is running inside Pyodide 0.25, I also checked Python's os.stat of the file, which reports the correct file size. Reading the file in Python also succeeds without the enormous over-allocation. It seems that this is a Rust-specific bug in getting the correct file size information from Emscripten.

Meta

My Rust toolchain is pinned to nightly-2024-07-21, the last nightly for 1.81.

[workingjubilee]

uhh does this reproduce on the latest nightly?

[juntyr]

uhh does this reproduce on the latest nightly?

Yes, unfortunately it still reproduces on rustc 1.83.0-nightly (eb4e23467 2024-10-09).

My current workaround is the following:

- let bytes = std::fs::read(path)?;
+ struct FileNoSize(std::fs::File);
+
+ impl std::io::Read for FileNoSize {
+     fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
+         self.0.read(buf)
+     }
+ }
+
+ let mut file = FileNoSize(std::fs::File::open(path)?);
+
+ let mut bytes = Vec::new();
+ std::io::Read::read_to_end(&mut file, &mut bytes)?;

[workingjubilee]

It seems we just call fstat64? It's not clear how that's not working out for us.

[workingjubilee]

We could have an incorrect definition of the Emscripten stat64 structure?

[kadiwa4]

emscripten's definition of stat64 changed in version 3.1.42, which is accounted for in the libc crate. Maybe that can lead to version-mismatch problems? Which version are you using? From this file, it seems like the Rust standard library still targets 2.0.5.

[juntyr]

Pyodide v0.25 uses Emscripten 3.1.46, so that would be after the change. The fix to libc was in rust-lang/libc@63b0d67 and released in v0.2.148, which Rust is already using. Huh.

[kadiwa4]

When building with a fixed version of libc, it does a check in the build script to see if the installed version of emscripten uses the old or new definition. I assume you downloaded the emscripten target from rustup, which would mean that you're using a precompiled standard library. So the question is really which version of emscripten is installed on the builders that built the standard library you are using.

[juntyr]

Do you think that using -Zbuild-std would help here?

[kadiwa4]

Sure, give it a try!