(Django) RabbitMQ + Celery

2024. 1. 22. 23:43ㆍ장고/파이썬 환경 설정

먼저 Celery 연동을 위해

pip install celery

pip install django-celery-results

pip install django-celery-beat

설치를 해주고 파일 설정을 해준다.

# 프로젝트/Settings.py

INSTALLED_APPS = [
	  ...
    'django_celery_beat'
    'django_celery_results'
		...
]

# Celery
CELERY_BROKER_URL = 'pyamqp://{ID}:{PWD}@localhost:5672' # 로컬용 -> 도커로 뭐 바꿔야 하는건가?
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
CELERY_RESULT_BACKEND = 'django-db'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

# 프로젝트/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab

# 기본 장고파일 설정
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '프로젝트명.settings')
app = Celery('프로젝트명')
app.config_from_object('django.conf:settings', namespace='CELERY')

#등록된 장고 앱 설정에서 task 불러오기
app.autodiscover_tasks()

# 앱/tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task

from dnd_7th_4_backend.celery import app

# test 용 함수
@shared_task
def printTime():
    print("Testtime: ", datetime.now())

자 여기서 rabbitmq-server를 실행 시켜주면 되는데 없는 명령어? 라고 뜰것이다… 설치해주자

.zshrc 에서

# rabbitmq
export PATH="$PATH:/usr/local/sbin"

근데 이거 환경변수 나는 이거 설정안해도 되던데 이유는 잘 모르겠습니다 → 찾아보기

brew update
brew install rabbitmq

명령어 실행해서 설치해주면 된다.

기본 포트: 15672
포그라운드 실행: 위 설치 위치에서 rabbitmq-server를 실행
백그라운드 실행: brew services restart rabbitmq

정도만 알면 된다고 한다.

포그라운드 실행

그럼 이제 rabbitmq-server를 입력하면 실패하는데 위에서 처럼 환경변수 설정하면 됨 실행 됨.

백그라운드 실행

brew services restart rabbitmq 를 입력하면 백그라운드 실행

백그라운드 실행을 종료하고 싶다면? brew services stop rabbitmq

백그라운드 실행하던거 종료 안하고 포그라운드 실행을 해버리면 이런 오류가 뜸..

BOOT FAILED
===========
ERROR: could not bind to distribution port 25672, it is in use by another node: rabbit@localhost

2024-01-17 22:01:20.286255+09:00 [error] <0.135.0> 
2024-01-17 22:01:20.286255+09:00 [error] <0.135.0> BOOT FAILED
2024-01-17 22:01:20.286255+09:00 [error] <0.135.0> ===========
2024-01-17 22:01:20.286255+09:00 [error] <0.135.0> ERROR: could not bind to distribution port 25672, it is in use by another node: rabbit@localhost
2024-01-17 22:01:20.286255+09:00 [error] <0.135.0> 
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>     supervisor: {local,rabbit_prelaunch_sup}
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>     errorContext: start_error
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>     reason: {dist_port_already_used,25672,"rabbit","localhost"}
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>     offender: [{pid,undefined},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {id,prelaunch},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {restart_type,transient},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {significant,false},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {shutdown,5000},
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0>                {child_type,worker}]
2024-01-17 22:01:21.291217+09:00 [error] <0.135.0> 
2024-01-17 22:01:21.292432+09:00 [error] <0.133.0>   crasher:
2024-01-17 22:01:21.292432+09:00 [error] <0.133.0>     initial call: application_master:init/4
2024-01-17 22:01:21.292432+09:00 [error] <0.133.0>     pid: <0.133.0>
2024-01-17 22:01:21.292432+09:00 [error] <0.133.0>     registered_name: []

GUI 관리 페이지 가고싶으면 http://localhost:15672 접속

초기 접속은 아이디 비밀번호 둘 다 guest 라고 함. 이렇게 뜸.

여기서 사용자를 추가할 수 있음

CLI에서도 가능

사용자 추가 → rabbitmqctl add_user {id} {pw}

사용자에 대한 권한 설정 → rabbitmqctl set_user_tags {id} {권한}

유저 삭제 rabbitmqctl delete_user {id}

유저 목록 rabbitmqctl list_users

암튼

로컬에서 테스트 해보기

rabbitmq-server 입력해서 실행.

그 다음 python 로컬 터미널 두 개 열어주기.

한개의 터미널에 celery -A 프로젝트명 worker -l info 입력 해준다.

rabbitmq, selery 연동 rabbitmq의 서버 실행은 잘 되었는데..

celery -A backend  worker --loglevel=info

를 입력하고 난 후

[2024-01-18 14:17:05,242: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,246: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,247: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,250: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,250: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,253: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,254: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,256: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,257: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,260: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,261: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:17:05,264: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@localhost:5672//: Connection.open: (530) NOT_ALLOWED - access to vhost '/' refused for user 'user'.
Trying to reconnect...

[2024-01-18 14:17:05,264: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s')
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 330, in start
    self._restart_state.step()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/billiard/common.py", line 153, in step
    raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
billiard.exceptions.RestartFreqExceeded: 5 in 1s

이 에러들이 반복되면서 출력이 된다..

그래서

해결 방법:

아래 명령은 'guest' 사용자에게 모든 교환기(exchanges), 큐(queues) 및 라우팅(routing)에 대한 권한을 부여하는 명령이다.

rabbitmqctl set_permissions -p / guest ".*" ".*" ".*"

위의 명령어를 입력 후에 오류는 나지 않았고 실행은 잘 되었는데 경고 문자는 그대로 떴다

[2024-01-18 14:21:04,032: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:21:04,036: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2024-01-18 14:21:04,037: WARNING/MainProcess] /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2024-01-18 14:21:04,040: INFO/MainProcess] mingle: searching for neighbors
[2024-01-18 14:21:05,058: INFO/MainProcess] mingle: all alone
[2024-01-18 14:21:05,082: INFO/MainProcess] celery@gimjin-yong-ui-MacBookPro.local ready.

이 경고가 뜨는 이유는? ChatGPT에게 물어보니

경고 메시지 (CPendingDeprecationWarning):

Celery는 브로커 연결 재시도 구성 (broker_connection_retry)에 대해 경고를 표시합니다. Celery 6.0 이상에서는 이 설정이 더 이상 시작 시 연결 재시도를 결정하지 않습니다. 이를 유지하려면, broker_connection_retry_on_startup 설정을 **True**로 설정해야 합니다.

그렇다고 한다 settings.py 파일에서 설정을 해주자.

# Celery
CELERY_BROKER_URL = 'pyamqp://guest:guest@localhost:5672//'  # 로컬 테스트용
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
CELERY_RESULT_BACKEND = 'django-db'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

추가해주고 다시 한 번 실행을 했더니?

[2024-01-18 14:25:33,576: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2024-01-18 14:25:33,579: INFO/MainProcess] mingle: searching for neighbors
[2024-01-18 14:25:34,598: INFO/MainProcess] mingle: all alone
[2024-01-18 14:25:34,615: INFO/MainProcess] celery@gimjin-yong-ui-MacBookPro.local ready.

아무런 오류 없이 잘 뜨게 되었다!

중간에 짚고 가자면

맨 처음 터미널에서는 rabbitmq-server 실행했고,

두번째 터미널에서는 celery -A backend worker --loglevel=info 실행했고

세번째 터미널에서 python shell을 실행한다.

자 그럼 이제 세번째 터미널 python shell을 실행시킨 다음 task.py 파일에 있는 내용을 실행 시켜보자

# 앱/tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task

from dnd_7th_4_backend.celery import app

# test 용 함수
@shared_task
def printTime():
    print("Testtime: ", datetime.now())

잘 됐다.

그 다음 프로젝트 디렉터리의 __ init __.py 파일에 이렇게 입력은 해주자

from .celery import app as celery_app

__all__ = ['celery_app']

이제 celery.py 파일에 task 등록해보자

# celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab

# 기본 장고파일 설정
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings')
app = Celery('backend')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.conf.beat_schedule = {
    'printTime': { # 스케줄링 이름
        'task': 'book.tasks.printTime',
        'schedule': crontab(), # 인자 없으면 매 분마다 실행
    }
}
#등록된 장고 앱 설정에서 task 불러오기
app.autodiscover_tasks()

이렇게 설정하고~

첫번째 터미널에서는 rabbitmq-server

두번째 터미널은 celery -A backend worker --loglevel=info

세번째 터미널은 celery -A backend beat --loglevel=info

이렇게 해주면 된다…! 근데 이건 로컬에서 한거임.

나는 도커에서 하고싶음

자 도커에서 해보자

그럼.. 일단 requirements.txt 파일에 셀러리에 관한것들 추가를 해준다

...
celery==5.2.0
django-celery-beat
django-celery-results
...

진짜 계속 안 되어서 구글링해보기도 하고 지피티한테도 물어보다가 결국에는 피티씨가 모양만 해결해준 것 같다.

settings.py, celery.py, tasks.py, docker-compose.yml 파일을 보여주고 짜달라고 하니까 해줬다..

일단 CELERY_BROKER_URL 주소를 변경 해주었다.
# Celery CELERY_BROKER_URL = 'pyamqp://guest:guest@localhost:5672//' # 로컬 테스트용 CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True CELERY_RESULT_BACKEND = 'django-db' CELERY_ACCEPT_CONTENT = ['application/json'] CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json'
docker-compose.yml 파일에 이렇게 코드를 작성하라고 했다.

version: '3'

services:
  rabbitmq:
    image: "rabbitmq:3-management"
    ports:
      - "15672:15672"
      - "5672:5672"
    restart: always

  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: test
    ports:
      - "3306:3306"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      retries: 5
      timeout: 5s

  web:
    build: .
    depends_on:
      - db
      - rabbitmq
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    restart: always
    environment:
      WAIT_HOSTS: db:3306, rabbitmq:5672

  celery_worker:
    build: .
    command: celery -A backend worker --loglevel=info
    volumes:
      - .:/app
    depends_on:
      - web
      - rabbitmq
    restart: always

  celery_beat:
    build: .
    command: celery -A backend beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - .:/app
    depends_on:
      - web
      - rabbitmq
    restart: always

빨간색 코드들을 작성해주면 됨.

그 다음에 docker-compose up 을 해줬더니 원래는 저 celery_beat-1 이란 애가 자꾸 꺼진다.

그래서 에러를 확인해보니 test(내 db table이름)에 어떤 db가 없다고하는 아래와 같은 오류가 떠서

**| [2024-01-19 02:08:24,402: WARNING/MainProcess] ^
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess]   File "/usr/local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess]     
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] db.query(q)
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess]   File "/usr/local/lib/python3.12/site-packages/MySQLdb/connections.py", line 255, in query
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess]     
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] _mysql.connection.query(self, query)
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] django.db.utils
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] .
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] ProgrammingError
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] : 
celery_beat-1    | [2024-01-19 02:08:24,402: WARNING/MainProcess] (1146, "Table 'test.django_celery_beat_periodictask' doesn't exist")
celery_beat-1    | celery beat v5.2.0 (dawn-chorus) is starting.
celery_beat-1    | __    -    ... __   -        _
celery_beat-1    | LocalTime -> 2024-01-19 02:08:24
celery_beat-1    | Configuration ->
celery_beat-1    |     . broker -> amqp://guest:**@rabbitmq:5672//
celery_beat-1    |     . loader -> celery.loaders.app.AppLoader
celery_beat-1    |     . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
celery_beat-1    | 
celery_beat-1    |     . logfile -> [stderr]@%INFO
celery_beat-1    |     . maxinterval -> 5.00 seconds (5s)
celery_beat-1 exited with code 1**

바로 web 콘솔에서 python manage.py migrate를 입력해줬더니 해결이 되었다.. (사실 도커 컨테이너 올렸으니까 당연히 했어야 하는건데 까먹었다.)

'장고 > 파이썬 환경 설정' 카테고리의 다른 글

파이썬 django drf-yasg 스웨거 연동 (0)	2024.01.13

파이썬 django drf-yasg 스웨거 연동 2024.01.13

개발자 지망생