server: (router) move model downloading to dedicated process

[ngxson]

Overview

Third item of #24822

To be merged after #24828

Currently, we spawn a downloading thread in router mode to download a new model. Now, router spawns a dedicated child process to handle that, making it the same as running a model (simplify the handling code)

TODO:

• better error handling
• clean up the temporary router->models entry
• wire up "download finished" event
• double check if "cancel" function still works correctly
• update dev docs
• test script: https://gist.github.com/ngxson/28b21fd0b7f0c532c28f1040b702955a

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: AI used for fixing some UB and to test it

[ngxson]

@ServeurpersoCom this affects the model management / downloading new model API, appreciate if you can do some quick tests!

[ServeurpersoCom]

Of course:)

[ServeurpersoCom]

Need rebase

[ServeurpersoCom]

Tested end to end, all good. Two small notes for UI: cancel and network failure both come back as download_failed, so a separate download_cancelled would read clearer,
And nit ? the progress events fire roughly every megabyte, which gets noisy on big models, so a little throttling would help

[ggerganov]

@ngxson This change caused a regression with my presets:

./bin/llama serve -hf ggerganov/preset-cuda-32gb 

0.00.785.635 I srv  llama_server: loading model
0.00.785.844 I srv    load_model: loading model ''
0.00.785.862 I common_init_result: fitting params to device memory ...
0.00.785.863 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.785.877 E llama_model_load_from_file_impl: exactly one out metadata, path_model, and file must be defined
0.00.785.947 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.785.954 E llama_model_load_from_file_impl: exactly one out metadata, path_model, and file must be defined
0.00.785.956 E common_init_from_params: failed to load model ''
0.00.785.958 E srv    load_model: failed to load model, ''
0.00.785.958 I srv    operator(): operator(): cleaning up before exit...
0.00.787.146 E srv  llama_server: exiting due to model loading error

[ngxson]

@ggerganov hmm, quite strange, it still works for me. I tried both:

./build/bin/llama serve -hf ggml-org/test-preset-ci
./build/bin/llama serve -hf ggerganov/preset-cuda-32gb

could you double-check if the build is up-to-date?

[ggerganov]

Yes, it's up to date:

 11:53:30 [141]  ./bin/llama version
b9767-7c908502e

 11:53:37  ./bin/llama serve -hf ggerganov/preset-cuda-32gb
0.00.059.994 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.059.998 I device_info:
0.00.060.001 I   - MTL0    : Apple M2 Ultra (165150 MiB, 165150 MiB free)
0.00.060.002 I   - BLAS    : Accelerate (0 MiB, 0 MiB free)
0.00.060.007 I   - CPU     : Apple M2 Ultra (196608 MiB, 196608 MiB free)
0.00.060.020 I system_info: n_threads = 16 (n_threads_batch = 16) / 24 | MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | ACCELERATE = 1 | REPACK = 1 | 
0.00.060.021 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.060.036 I srv          init: running without SSL
0.00.060.073 I srv          init: using 23 threads for HTTP server
0.00.529.242 I srv         start: binding port with default address family
0.00.530.614 I srv  llama_server: loading model
0.00.530.650 I srv    load_model: loading model ''
0.00.530.663 I common_init_result: fitting params to device memory ...
0.00.530.664 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.530.672 E llama_model_load_from_file_impl: exactly one out metadata, path_model, and file must be defined
0.00.530.715 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.530.873 E llama_model_load_from_file_impl: exactly one out metadata, path_model, and file must be defined
0.00.530.876 E common_init_from_params: failed to load model ''
0.00.530.877 E srv    load_model: failed to load model, ''
0.00.530.878 I srv    operator(): operator(): cleaning up before exit...
0.00.531.296 E srv  llama_server: exiting due to model loading error

I think the issue is that this logic runs too early now:

llama.cpp/tools/server/server.cpp

Lines 92 to 94 in 7c90850

	// router server never loads a model and must not touch the GPU
	const bool is_router_server = params.model.path.empty()
	&& params.model.hf_repo.empty();

The preset detection runs in common_params_handle_models() which now runs later.

[ngxson]

ah yeah, you're right, my branch was not up-to-date.

having a look now!