posthog/bin/celery-queues.env

# Default set of queues to be used by Celery.
# Important: Add new queues to make Celery consume tasks from them.

# NOTE: Keep in sync with posthog/tasks/utils.py
CELERY_WORKER_QUEUES=celery,stats,email,insight_export,insight_refresh,analytics_queries,exports,subscription_delivery,usage_reports,session_replay_embeddings
feat(celery): Prepare to run on multiple queues (#19157) * Add Celery queues env file with default queues Reasoning: We need to configure Celery workers in several places to consume from a specific set of queues. * Define some queues 2024-01-17 12:54:12 +01:00			`# Default set of queues to be used by Celery.`
			`# Important: Add new queues to make Celery consume tasks from them.`
feat: Celery tasks refactor (#19827) 2024-01-18 14:28:10 +01:00
			`# NOTE: Keep in sync with posthog/tasks/utils.py`
feat: generate recording text embeddings (#20046) * make migration * general flow * abstract shared methods * generate input * remove postgres migration * generate embedding strings * remove random file * Update query snapshots * Update query snapshots * feat: create periodic replay embedding * first sketch of table * batch and flush embeddings * add default to timestamp generation * fetch recordings query * save first embeddings to CH * dump session metadata into tokens * fix lint * brain dump to help th future traveller * prom timing instead * fix input concatenation * add an e :/ * obey mypy * some time limits to reduce what we query * a little fiddling to get it to run locally * paging and counting * Update query snapshots * Update query snapshots * move the AI stuff to EE for now * Update query snapshots * kick off the task with interval from settings * push embedding generation onto its own queue * on a different queue * EE to the max * doh * fix * fangling * Remove clashes so we can merge this into the other PR * Remove clashes so we can merge this into the other PR * start wiring up Celery task * hmmm * it's a chord * wire up celery simple version * rename * why is worker failing * Update .run/Celery.run.xml * update embedding input to remove duplicates * ttl on the table * Revert "update embedding input to remove duplicates" This reverts commit 9a09d9c9f0c841f42abf0853b38b5922b67f7d23. --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Paul D'Ambra <paul@posthog.com> 2024-02-14 13:50:42 +01:00			`CELERY_WORKER_QUEUES=celery,stats,email,insight_export,insight_refresh,analytics_queries,exports,subscription_delivery,usage_reports,session_replay_embeddings`