Celery

Celery is an open-source distributed task queue for Python applications that enables asynchronous task processing, background job execution, and scheduled workflows. It is widely used by developers, DevOps teams, data engineers, and organizations building scalable web applications, microservices, automation systems, and distributed processing workloads.

Key Features of Celery:

• Provides asynchronous and distributed task execution for Python applications.
• Processes background jobs outside the main application workflow to improve responsiveness.
• Supports scheduled and periodic task execution for automated workflows.
• Integrates with message brokers and backends such as Redis and RabbitMQ.
• Supports task retries, error handling, and distributed worker processing for resilient workloads.
• Enables scalable background processing across multiple worker processes and systems.

Celery usage:

Celery usage:

Open Terminal 1:

$ sudo su
$ cd /opt
$ cd ~/celery-app
$ source venv/bin/activate

Check the installed Celery version:
$ celery --version

Check the Celery Python package version:
$ python -c "import celery; print(celery.__version__)"

Check Celery package details:
$ pip show celery

Start the Celery worker:
$ celery -A celery_app worker --loglevel=info


Open Terminal 2:

Connect to the Celery server:
$ ssh YOUR_USERNAME@YOUR_SERVER_IP

Activate the Celery virtual environment:
$ cd ~/celery-app
$ source venv/bin/activate

Test the Celery task:
$ python

Run:
$ from celery_app import add
$ result = add.delay(10, 20)
$ print(result.get(timeout=10))

Expected output:
$ 30

Check the Celery worker processes:
$ ps aux | grep celery

Deactivate the virtual environment:
$ deactivate
  

Disclaimer:
Celery is an open-source project and is not affiliated with or endorsed by Microsoft Azure or any other cloud provider. Users are responsible for configuring message brokers, result backends, network access, authentication, security settings, firewall rules, application configuration, and worker processes when deploying and using Celery in a cloud environment.