I'm new to using Celery with Django and using RabbitMQ as a message broker, and I'm having trouble being able to get a progress bar of sorts working.
I have a function in the views.py, called running_tagging, which runs a celery task, gpt_manager. With my current setup, the docker containers for RabbitMQ, the celery worker and the django app all spin up without a problem. I can also run the view that triggers the celery task successfully, and see in the celery logs that the job is proceeding as it should. However, the responses that come back when I'm looking at the Network tab in dev tools do not show any updates, and the django admin panel also doesn't show the new task result etc.
I'm not 100% sure whether it's an issue, but I've also had trouble with understanding whether I need django-celery or just celery for this, and have been having some issues with version compatibility when trying to install both. If it's any help these are the versions of any packages related to django and celery that I have installed currently:
- celery==3.1.26.post2
- celery-progress==0.3
- Django==4.2.7
- django-celery==3.3.1
- django-celery-results==2.5.1
here is the function from my views.py file:
obj_pk = request.session.get('theme_finder_object_pk')
obj_pk = UUID(obj_pk)
tfObj = ThemeFinderExercise.objects.get(id=obj_pk)
batches_list = request.session.get('batches', [])
batches_df_list = [pd.read_json(df_json) for df_json in batches_list]
batches_dict_list = [df.to_dict(orient='records') for df in batches_df_list]
result = gpt_manager.delay(batches_dict_list, 2000, tfObj.content_col, tfObj.metaprompt)
context = {'task_id':result.id}
return render(request, 'themefinder/theme-finder-run-tagging.html', context)
This is my tasks.py file:
@shared_task(bind=True)
def gpt_manager(self, batches, max_tokens, content_col_name, meta_prompt):
progress_recorder = ProgressRecorder(self)
results = []
# Create empty lists to store extracted data
themes = []
descriptions = []
example_comments = []
batch_num = []
num_batches = len(batches)
counter = 1
print('about to start batch 1')
for batch in batches:
df = pd.DataFrame(batch)
concatenated_text = '\t'.join(df[content_col_name].astype(str))
print('passing into api call')
result = gpt_theme_finder(concatenated_text, meta_prompt, max_tokens)
print('api call result:')
print(result)
lines = result.split('\n')
# Loop through the list and extract information
current_theme = ""
current_description = ""
current_example_comment = ""
for line in lines:
batch_num.append(counter)
if line.startswith('Theme'):
current_theme = line.split(': ', 1)[1]
themes.append(current_theme)
elif line.startswith('Description'):
current_description = line.split(': ', 1)[1]
descriptions.append(current_description)
elif line.startswith('Example comment'):
# Check if there is a colon (:) after 'Example comment'
if ': ' in line:
current_example_comment = line.split(': ', 1)[1]
example_comments.append(current_example_comment)
else:
# If no colon, then the whole line is the example comment
current_example_comment = line.split('Example comment: ')[1]
example_comments.append(current_example_comment)
progress_recorder.set_progress(counter, num_batches)
print(counter, "batch processed of ", len(batches))
counter = counter +1
This is my html file:
{% extends 'main.html' %}
{% load static %}
{% block content %}
<head>
<style>
.progress-wrapper {
width: 50%;
background-color: white !important;
}
</style>
</head>
<h1>Theme Finder</h1>
<div class="bg-light rounded h-80 p-4 w-60">
<h3 class="mb-4">Tagging in Progress</h3>
<div class="mb-3" id="message">
<p>Please don't navigate away from this page while tagging is in process</p>
</div>
{% csrf_token %}
<div class='progress-wrapper'>
<div id='progress-bar' class='progress-bar bg-info'> </div>
</div>
<div id="progress-bar-message">Waiting for progress to start...</div>
<button type="submit" class="btn btn-primary btn-danger m-2" id="back-cancel"><i class="fa fa-angle-double-left"></i> Back</button>
<button type="submit" class="btn btn-primary btn-success m-2" id="continue-output">View Output <i class="fa fa-angle-double-right"></i></button>
</div>
<script src="{% static 'celery_progress/celery_progress.js' %}"></script>
<script>
document.addEventListener("DOMContentLoaded", function () {
var progressUrl = "{% url 'celery_progress:task_status' task_id %}";
// var progressBar = document.getElementById('progress-bar');
var outputButton = document.getElementById('continue-output');
CeleryProgressBar.initProgressBar(progressUrl);
outputButton.addEventListener('click', function () {
window.location.href = "{% url 'tf-output' %}";
});
});
</script>
{% endblock %}
These are the relevant parts of my settings.py file:
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_celery_results',
'celery_progress',
'djcelery',
'landing.apps.LandingConfig',
'userprofiles.apps.UserprofilesConfig',
'themefinder.apps.ThemefinderConfig',
'projects.apps.ProjectsConfig',
]
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_TRACK_STARTED = True
CELERY_BROKER_URL= "pyamqp://guest:guest@rabbitmq:5672//"
import djcelery
djcelery.setup_loader()
I don't know how relevant it will be, but I've also included my docker-compose file below:
version: '3.8'
services:
python:
&python
build:
context: .
dockerfile: Dockerfile
environment:
- DEBUG=1
- ALLOWED_HOSTS=*
- PYTHONUNBUFFERED=1
- PYTHONDONTWRITEBYTECODE=1
- CELERY_BROKER_URL=pyamqp://guest:guest@rabbitmq:5672//
- CELERY_RESULT_BACKEND=djcelery.backends.database:DatabaseBackend
ports:
- 8000:8000
command: /usr/src/app/entrypoint-django.sh
depends_on:
- rabbitmq
- celery_worker
rabbitmq:
image: rabbitmq:3.7-alpine
celery_worker:
build:
context: .
dockerfile: Dockerfile
command: /usr/src/app/entrypoint-celery.sh
ports: []
depends_on:
- rabbitmq
I can't think of anything else that might be helpful to put here, but let me know if there's anything and I'll make an update and add it. Hoping someone can help me, just want to get this progress bar going!