Feature/celery new execute job

[Prajna1999]

Summary

Target issue is #PLEASE_TYPE_ISSUE_NUMBER
Explain the motivation for making this change. What existing problem does the pull request solve?

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

Summary by CodeRabbit

Release Notes

• Refactor

  • Restructured background job processing system to use dedicated task dispatchers for each job type, replacing a generic dispatch mechanism.
  • Enhanced job tracing with improved correlation ID tracking for better visibility into asynchronous operations.

• Chores

  • Updated worker process recycling policy for more frequent child process replacement.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 70df26cc-0e33-4586-ac34-d9960a53abd3

📥 Commits

Reviewing files that changed from the base of the PR and between e620c9b and b66eeda.

📒 Files selected for processing (13)

• backend/app/api/routes/stt_evaluations/evaluation.py
• backend/app/api/routes/tts_evaluations/evaluation.py
• backend/app/celery/celery_app.py
• backend/app/celery/tasks/job_execution.py
• backend/app/celery/utils.py
• backend/app/core/config.py
• backend/app/crud/stt_evaluations/cron.py
• backend/app/crud/tts_evaluations/cron.py
• backend/app/services/collections/create_collection.py
• backend/app/services/collections/delete_collection.py
• backend/app/services/doctransform/job.py
• backend/app/services/llm/jobs.py
• backend/app/services/response/jobs.py

---

📝 Walkthrough

Walkthrough

This pull request refactors the Celery task dispatch system by replacing generic function-path-based job starters with dedicated, type-specific task helpers. All call sites are updated to use explicit starters like start_stt_batch_submission, start_llm_job, and start_tts_batch_submission. The worker child process recycling is made more aggressive, and trace correlation IDs are integrated into task execution.

Changes

Cohort / File(s)	Summary
Celery Task Dispatch Infrastructure backend/app/celery/utils.py, backend/app/celery/tasks/job_execution.py, backend/app/celery/celery_app.py	Removed generic start_high_priority_job and start_low_priority_job functions (which used function_path dispatch). Added 10 dedicated starters (start_llm_job, start_stt_batch_submission, start_tts_batch_submission, etc.) and corresponding Celery tasks (run_llm_job, run_stt_batch_submission, etc.). Integrated _set_trace() helper to set correlation IDs. Removed function registry and task routing entries; explicit queue/priority config moved to task decorators.
API Route Updates backend/app/api/routes/stt_evaluations/evaluation.py, backend/app/api/routes/tts_evaluations/evaluation.py	Updated route handlers to call start_stt_batch_submission and start_tts_batch_submission instead of generic start_low_priority_job with function paths.
Service Job Starters backend/app/services/llm/jobs.py, backend/app/services/response/jobs.py, backend/app/services/doctransform/job.py, backend/app/services/collections/create_collection.py, backend/app/services/collections/delete_collection.py	Replaced generic job starters with dedicated helpers: start_llm_job, start_response_job, start_doctransform_job, start_create_collection_job, start_delete_collection_job. Removed explicit function_path arguments.
Cron Task Dispatch backend/app/crud/stt_evaluations/cron.py, backend/app/crud/tts_evaluations/cron.py	Updated cron job dispatch to use start_stt_metric_computation and start_tts_result_processing instead of generic start_low_priority_job with hardcoded function paths.
Configuration backend/app/core/config.py	Reduced CELERY_WORKER_MAX_TASKS_PER_CHILD from 1000 to 1, enabling more aggressive worker child process recycling.

Sequence Diagram(s)

sequenceDiagram
    participant API as API Route
    participant Starter as Task Starter<br/>(start_stt_batch_submission)
    participant Celery as Celery Broker
    participant Worker as Worker
    participant Tracer as Trace Manager<br/>(_set_trace)
    participant Service as Service<br/>(execute_batch_submission)

    API->>Starter: start_stt_batch_submission(project_id, job_id, trace_id, ...)
    Starter->>Celery: run_stt_batch_submission.delay(...)
    Celery-->>Worker: Task enqueued (high_priority queue)
    Worker->>Tracer: _set_trace(trace_id)
    Tracer->>Tracer: Set asgi_correlation_id
    Worker->>Service: execute_batch_submission(project_id, job_id, task_id, ...)
    Service-->>Worker: Result
    Worker-->>API: Task ID returned

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Hotfix/celery import issues #705: Modifies the same Celery task dispatch layer (celery_app.py, tasks/job_execution.py, utils.py), directly overlapping with core infrastructure changes.
• STT Evaluation: Refactor #601: Introduces STT batch submission executor; this PR integrates it into the new dedicated task/starter pattern (run_stt_batch_submission / start_stt_batch_submission).
• STT Evaluation: Automated metric #639: Adds STT metric computation task; this PR switches cron dispatch from generic to dedicated start_stt_metric_computation helper.

Suggested labels

enhancement, ready-for-review

Suggested reviewers

• kartpop
• vprashrex

Poem

🐰 Through celery fields, a bunny hops with glee,
No more generic paths—just tasks wild and free!
Traces now gleam, correlation IDs bright,
Workers recycle faster, priorities set right!
From chaos to clarity, the refactor complete. ✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/celery-new-execute-job

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}