MPI for Darwin

[chadbrewbaker]

Added minimal support to compile MPI on Darwin.

TODO

• Investigate tensor operations that benefit from cache locality of MPI parallelism.
• Investigate MPI_Scan for token evaluation.
• Investigate MPI_IO for faster model loading.
• Investigate MPI playing nice with Apple Neural Engine, Metal Shaders, and possible abuse of video codec hardware

[chadbrewbaker]

@steveklabnik I'm actually writing this with an @oxidecomputer rack in mind similar to a Blue Gene/L. What MPI upstream distro should I use for Linux?

[prusnak]

I think it has been said by @ggerganov somewhere that he doesn't want to introduce MPI or other more complex dependencies into the project. I'm not able to find the reference though at the moment.

[chadbrewbaker]

I think it has been said by @ggerganov somewhere that he doesn't want to introduce MPI or other more complex dependencies into the project. I'm not able to find the reference though at the moment.

I threw it under a LLAMA_MPI_DARWIN #ifdef to keep it out of the way. Zero cost abstraction unless you need to scale.

[ggerganov]

I'm not familiar with MPI, but feel free to investigate and add support for it.
It will almost certainly not be part of ggml, but it can be a llama.cpp example demonstrating the integration with this MPI platform / framework

[evanmiller]

Checkout #2099 for a working MPI proof of concept

[chadbrewbaker]

Checkout #2099 for a working MPI proof of concept

Will do! Three areas for parallelism. MPI IO, distributed memory on single passes, and I think there is a parallel prefix formulation.

Good talk at Stanford’s ML seminar a few months ago on compressed representations over the wire when doing distributed memory for training. Assuming inference will also allow on the fly compressed representations but I need to think about it more.

[ggerganov]

Superseded by #2099