mirror of
https://github.com/PostHog/posthog.git
synced 2024-11-24 18:07:17 +01:00
bbd41ef04a
From https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb > By default, preforking Celery workers distribute tasks to their worker processes as soon as they are received, regardless of whether the process is currently busy with other tasks. > If you have a set of tasks that take varying amounts of time to complete — either deliberately or due to unpredictable network conditions, etc. — this will cause unexpected delays in total execution time for tasks in the queue. This is 100% the case for us. This should "load balance" the tasks better across workers.
12 lines
325 B
Bash
Executable File
12 lines
325 B
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
|
|
# this kills all processes when the last one terminates
|
|
trap 'kill $(jobs -p)' EXIT
|
|
|
|
# start celery worker with heartbeat (-B)
|
|
celery -A posthog worker -B --scheduler redbeat.RedBeatScheduler --without-heartbeat --without-gossip --without-mingle -Ofair &
|
|
|
|
# start celery plugin worker
|
|
./bin/plugin-server
|