STGL update blog

  • misc
  • Updates

SQS setup for celery applications

Published: Tue 18 April 2017
By Robert Sare

In misc.

tags: AWS Python

This is a summary of steps needed to get celery working with Amazon S3 bucket storage and the Simple Queue Service.

  1. The SQS broker works with boto <= 2.9.2, celery <= 3.1.24, kombu <= 3.0.37.

    • To use S3 as a result backend, you need to install celery-s3 and specify CELERY_RESULT_BACKEND = 'celery_s3.backends.S3Backend'

  2. A user with S3 access must be created, and the user's access key and ID must be used in BROKER_URL and CELERY_S3_BACKEND_SETTINGS. (As opposed to the AWS access key/ID or other authentication credentials.)

    • The queue tself can use the SQSDefaultPolicy permissions.

  3. CELERY_DEFAULT_QUEUE and queue_name_prefix (in BROKER_TRANSPORT_OPTIONS) must be set.

    • If no default queue is specified, a celery queue will be created. This may or may not show up in the SQS queue monitor.
    • If no prefix is specified, individual workers will not create their own pidbox'ed queues.
Proudly powered by Pelican, which takes great advantage of Python.

The theme is by Smashing Magazine, thanks!