Move `stop_runner()` from `process_runs` to `process_terminating_jobs`

[r4victor]

Related to #3551

process_runs is a background task that updates run/job statuses, performs scaling, etc, and it's quick in all cases except when processing terminating runs – it terminates runs' active jobs by marking them as terminating and also calling stop_runner(), which communicates with the runner and thus can be slow.

stop_runner() should be moved to process_terminating_jobs. process_run should only mark jobs as needed graceful termination. This is primarily needed for #3551 pipeline processing, which is efficient when pipeline's processing times do not vary much.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[r4victor]

Another issue with the way stop_runner() is called now – it's done sequentially in a loop, which is slow when there are many running jobs, e.g. a run with 100 running jobs.

dstack/src/dstack/_internal/server/background/pipeline_tasks/runs/terminating.py

Lines 75 to 86 in 75ff17a

	for job_model in locked_jobs:
	if job_model.status == JobStatus.RUNNING and job_termination_reason not in {
	JobTerminationReason.ABORTED_BY_USER,
	JobTerminationReason.DONE_BY_RUNNER,
	}:
	# Send a signal to stop the job gracefully.
	await stop_runner(
	job_model=job_model, instance_model=get_or_error(job_model.instance)
	)
	delayed_job_ids.append(job_model.id)
	continue
	regular_job_ids.append(job_model.id)