Optimize Job Processing Endpoint to Use Async Queue

[Mohiiit]

Description:

The current implementation of the handle_process_job_request endpoint directly calls the process_job function, which might involve long-running operations. Since this endpoint is part of a REST API, prolonged processing times can lead to timeouts, degraded performance, and a poor user experience.

Current Code Snippet:

#[instrument(skip(config), fields(job_id = %id))]
async fn handle_process_job_request(
    Path(JobId { id }): Path<JobId>,
    State(config): State<Arc<Config>>,
) -> JobRouteResult {
    let job_id = Uuid::parse_str(&id).map_err(|_| JobRouteError::InvalidId(id.clone()))?;

    match process_job(job_id, config).await {
        Ok(_) => {
            info!("Job processed successfully");
            ORCHESTRATOR_METRICS.successful_job_operations.add(1.0, &[KeyValue::new("operation_type", "process_job")]);
            Ok(Json(ApiResponse::success()).into_response())
        }
        Err(e) => {
            error!(error = %e, "Failed to process job");
            ORCHESTRATOR_METRICS.failed_job_operations.add(1.0, &[KeyValue::new("operation_type", "process_job")]);
            Err(JobRouteError::ProcessingError(e.to_string()))
        }
    }
}

Problem Statement:

• Long-running operations in an HTTP endpoint can result in timeouts.
• The API should respond quickly after successfully adding the job to a queue.

Proposed Solution:

• Instead of directly processing the job in the HTTP handler, add the job to an asynchronous task queue.
• Return a 200 OK response immediately after successfully adding the job to the queue.