[Bug]: FAILED_TO_START_DUE_TO_NO_CAPACITY

[judeleonard]

Steps to reproduce

create a fleet.dstack.yml to provision remote backend from my on_prem server. This was created successfully.

dstack apply -f fleet.dstack.yml

type: fleet
name: model-dev-fleet

placement: any

# The user, private SSH key, and hostnames of the on-prem servers
ssh_config:
  user: my_user
  identity_file: ~/.ssh/id_rsa
  hosts:
    - 33.33.48.1

create another yml config to provision a development server with the provisioned fleet as the backend

dstack apply -f dev_environment.yml

type: dev-environment
name: model-dev-env

#python: "3.11"  
image: dstackai/base:py3.13-0.6-cuda-12.1

ide: vscode

spot_policy: auto

Actual behaviour

I got the below error

All provisioning attempts failed. This is likely due to cloud providers not having enough capacity. Check CLI and server logs for more details.

Then also tried to see extra details about the error with the below command

dstack ps --verbose

output below

NAME           BACKEND  REGION  INSTANCE  RESOURCES  SPOT  PRICE  STATUS  SUBMITTED   ERROR                                
 model-dev-env                                                     failed  54 sec ago  JOB_FAILED                           
                                                                                       (FAILED_TO_START_DUE_TO_NO_CAPACITY) 

Expected behaviour

Instance provisioning should be completed successfully with a vscode link to my workspace.

dstack version

0.18.22

Server logs

[12:45:55] INFO     dstack._internal.server.services.backends:404 Requesting instance offers from backends: []                                                                                              
[12:45:56] INFO     dstack._internal.server.background.tasks.process_runs:330 run(110058)model-dev-env: run status has changed SUBMITTED -> TERMINATING                                                     
           INFO     dstack._internal.server.services.jobs:283 job(0d49c8)model-dev-env-0-0: job status is FAILED, reason: FAILED_TO_START_DUE_TO_NO_CAPACITY                                                
[12:45:58] INFO     dstack._internal.server.services.runs:739 run(110058)model-dev-env: run status has changed TERMINATING -> FAILED, reason: JOB_FAILED

Additional information

No response

[peterschmidt85]

@judeleonard please provide dstack fleet list output.

One of the reasons can be that the fleet instance have GPUs while the dev environment doesn't request any.

[judeleonard]

Here is the output

 FLEET            INSTANCE  BACKEND       RESOURCES                   PRICE  STATUS      CREATED     
 model-dev-fleet  0         ssh (remote)  2xCPU, 0GB, 100.0GB (disk)  $0.0   terminated  3 hours ago 

[judeleonard]

@judeleonard please provide dstack fleet list output.

One of the reasons can be that the fleet instance have GPUs while the dev environment doesn't request any.

Yes, I tried to attach a GPU before but I got the same error 'Not having enough capacity' And my remote server actually has both GPU and docker preinstalled

[peterschmidt85]

Here is the output

It's dstack ps, not dstack fleet list

dstack by default offers only instances that match exactly the resources of the fleet

[judeleonard]

This is the output. Not much details

NAME           BACKEND  REGION  RESOURCES  SPOT  PRICE  STATUS  SUBMITTED   
 model-dev-env                                           failed  31 mins ago 

[peterschmidt85]

Means the fleet creation wasn't successful.

1. Please try again to create the fleet, and post here the entire output.
2. After that, also please post here the output of dstack server

This will help understand why fleet cound't be created

[judeleonard]

This is the dastack webserver after the fleet was created. But I will try again and post the entire output

[judeleonard]

I just recreated the fleet now

 dstack apply -f fleet.dstack.yml 

/usr/lib/python3/dist-packages/paramiko/transport.py:237: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
 Project        main             
 User           admin            
 Configuration  fleet.dstack.yml 
 Type           fleet            
 Fleet type     ssh              
 Nodes          1                
 Placement      any              

Found fleet model-dev-fleet. Configuration changes detected.
Re-create the fleet? [y/n]: y

 FLEET            INSTANCE  BACKEND       RESOURCES  PRICE  STATUS   CREATED     ERROR 
 model-dev-fleet  0         ssh (remote)             $0.0   pending  20 sec ago        

then my stack server log.

Could this fingerprint be an issue with my ssh user?

                 'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:13] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
[16:00:14] WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:19] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:24] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
[16:00:25] WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:29] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:34] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
[16:00:35] WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:39] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:43] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:48] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:52] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:00:57] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:01:03] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:01:08] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                                                                          
[16:01:13] INFO     dstack._internal.server.background.tasks.process_instances:217 Adding ssh instance model-dev-fleet-0...                                                                                 
          WARNING  dstack._internal.server.background.tasks.process_instances:281 Provisioning instance model-dev-fleet-0 could not be completed because of the error: Deploy instance raised an error:    
                   'RSAKey' object has no attribute 'fingerprint'                                                                                  

[peterschmidt85]

Oh that's a known issue, it will be fixed in the next release but for now please do pip install paramiko -U and then restart the server, and try again. The issue must be gone

[judeleonard]

Thank you. Will do that

[judeleonard]

hi @peterschmidt85 , sorry for updating you at this time. Our remote server was undergoing some updates.

So I later tried it after installing paramiko like you suggested and dstack server logs indeed changed from what I had before. This is my log now. I also updated dstack to the latest v0.18.25.

[peterschmidt85]

@judeleonard Now that you've updated paramiko, please show the full output of creating the fleet (both dstack apply and dstack server outputs).