feat: WIP: Adjust GPU Layers

[siddimore]

Description

1. Add GGUF Parser

TODO

1. Install nvidia-smi driver
2. Fetch GPU Device information
3. Offload layers to GPU baed on GGUF Parser metadata

This PR fixes #3541

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	cd1dc5d
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6705d246290228000812ee36
😎 Deploy Preview	https://deploy-preview-3737--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[siddimore]

@mudler can u kindly check the PR approach and give some high level feedback when possible?

Next step that i will add is some sort of GPU_Layer estimator based on:

1. VRAM from GGUF Parsing
2. Noof GPU's on the device

[mudler]

Minor nit: I would call this something like AutomaticallyAdjustGPULayers :

Suggested change

	AdjustGPULayers bool `env:"LOCALAI_ADJUST_GPU_LAYERS,ADJUST_GPU_LAYERS" help:"Enable OffLoading of model layers to GPU" group:"models"`
	AutomaticallyAdjustGPULayers bool `env:"LOCALAI_AUTO_ADJUST_GPU_LAYERS,ADJUST_GPU_LAYERS" help:"Enable Automatic OffLoading of model layers to GPU" group:"models"`

[mudler]

Minor nit, but it's good practice to keep acronyms uppercase, e.g. UnmarshalYAML, GetXYZ:

Suggested change

	func GetNvidiaGpuInfo() ([]GPUInfo, error) {
	func GetNvidiaGPUInfo() ([]GPUInfo, error) {

[mudler]

This is still a minor nit, but all other tests are using ginkgo - do you feel to use ginkgo as well? not a blocker in any case, at some point will refactor things out to be more consistent if needed

[mudler]

Small nit style (non-blocking): it would make the code more reusable if the interface here would require only ParseGGUFFile and have different parsers implementing their ParseGGUFFile logic, for instance a Ollama parser, a Huggingface Parser, etc.

The caller then would need to instantiate the needed parser down the line, for instance:

type GGUFParser interface {
   Parse(path string) (*ggufparser.GGUFFile, error)
}

// GetModelGGufData returns the resources estimation needed to load the model.
func GetModelGGufData(modelPath string, estimator ModelMemoryEstimator, ollamaModel bool) (*ModelEstimate, error) {
	ctx := context.Background()

	fmt.Println("ModelPath: ", modelPath)

        var  ggufParser GGUFParser
 
	// Check if the input is a valid URL
        switch {
          case isURL(modelPath):
             ggufParser = &RemoteFileParser{ctx,modelPath}
           case ollamaModel:
             ggufParser = &OllaamParser{ctx,modelPath}
          /// .. other parsers here
        }
      return estimator.Estimate(ggufRemoteData)
}

Considering that we pass an estimator down the line tells me that this actually should be part of ModelMemoryEstimator as well:

func (g GGUFEstimator) GetModelGGufData(modelPath string,  ollamaModel bool) (*ModelEstimate, error) {

[mudler]

@siddimore thanks for taking a stab at this, direction looks good here - just few minor nits here and there but definitely not blockers

[siddimore]

@siddimore thanks for taking a stab at this, direction looks good here - just few minor nits here and there but definitely not blockers

thanks much @mudler you are welcome!! i will improve the code and add some more testing. Appreciate the feedback and will fix the comments

[github-actions]

This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been stalled for 10 days with no activity.