The right way to call a celery task from Django code?

Question

I have a registration form that upon submission, I want to perform some data validation via some external APIs, then eventually submit to another API. I want to do this asynchronously so the submitter is not waiting 10 seconds for the form to submit.

This is my code so far -- it works, but if an exception is thrown in the task, the user gets a 500 error (the specific exception is that sometimes a third party API I use times out). My impression was that if the task is executed asynchronously (notice the @task decorator and using delay to call the task), then the success page load should happen regardless of what happens in the function.

views.py

from myapp.tasks import prepare_and_submit_data

def register(request):
  form = SignupForm(request.POST or None)
  if form.is_valid():

    # Async function.
    prepare_and_submit_data.delay(form.cleaned_data)

    return render(request, 'success.html', {})
  return render(request, 'signup.html', {'form': form})

tasks.py

from celery.decorators import task

@task
def prepare_and_submit_data(data):
  # do some external API requests and modify data
  data = requests.post(BASE_URL, data=data).json()
  # ...

Is the right solution to wrap the contents of the prepare_and_submit_data with a try/except clause and log the error? Or wrap the calling of the function in register?

Thank you.

Answer 1

If the task is executed asynchronously, the call for delay is non-blocking, it just puts the task into the message queue and immediately continues with the execution of your view's code, which means no matter whether the task succeeds or fails, it can't have any effect on the view's code. Exception's to this statement are:

• if task queuing fails for some reason, e.g. your message queue is unreachable so the task cannot be scheduled, in that case you'll get an exception when calling prepare_and_submit_data.delay

• CELERY_ALWAYS_EAGER is set to True, which means that even when you're explicitly requesting a task to be executed asynchronously, it will be executed synchronously nevertheless, which is the same as if you were just calling a blocking function regularly (prepare_and_submit_data())

My bet is on the latter case, your have written the code properly, but the above mentioned setting forces it to be executed synchronously. One thing to consider though, it is almost sure that you won't have the results of the async task available when your view returns the response. So if you intended to use the result of the async validation during the execution of your view, then you just don't need to run it asynchronously, you were after regular synchronous execution from the beginning. But, if you need to use it only later, a common technique is to use periodical ajax requests from the client side towards a separate API endpoint, that should check whether your task finished it's execution and return the results.