mirrors/posthog
0
0
Fork 0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-22 08:40:03 +01:00
Code Issues Releases Activity
165ccf4924
posthog/bin/start-worker

18 lines
513 B
Plaintext
Raw Normal View History

scripts to run background, worker and frontend simultaneously
2020-04-07 12:34:01 +02:00
#!/bin/bash
set -e
Plugin v8 (#1946) * plugin progress * blah * add posthog config for plugins * test gitignore * new functionality for plugins * support local plugin paths * also ignore symlinks * add positional argument * fixes * small fixes * config polish * config passed to posthog plugin * ooooooops * symlink fix * cleanse dir before loading * add cache to plugins * pickle the goods * unlink symlink * pass full config * unlink even if link points to nothing * fix fix * return none if value is empty * plugin model * plugins scene * add config schema to plugins * install plugins * save descriptions * show descriptions * edit plugin * save plugin config * plugin modal * uninstall plugins * UX cleanup * add "required" to plugin config * open plugin modal after install * split to subcomponents * install custom plugins * rework backend for model plugins * Plugins on models * simple reload pubsub * fix apps not installed * fix master/main issue * fix reload command * use the github api to get the default branch * init plugins only if not running migrate/makemigrations * store plugins zip archives in postgres * tag plugins to specific versions * save plugins in pluginConfig * update pluginConfigs instead of adding new rows, remove from redux on uninstall * remove debug * run plugins from db by team * reload when deleting * remove debug * smarter handling of dynamic plugins, support local plugins again * improve typings, add some nicer warnings * yarn lock file after merge * squash migrations and add "locked" field to plugins * error if folder not found in zip * unregister plugins * skip plugin init in test mode * basic plugin test * avoid mutating the same prop hash * add pip tools to requirements.txt * fix mypy, fix manage.py script error * avoid plugins with mypy * mypy fix * abstract redis into plugin and add team_id to reload * refactor and start work on syncing with posthog.json * start testing plugin loading from json * test plugin deletion * test for syncing plugins from config * complete and then test local json plugin sync * test converting back and forward between an local and http path * remove global plugin config from plugins array in posthog.json * rename configSchema --> config_schema * fix migration after merge * rename from_cli to from_json * mypy * import pip after plugin loaded * show error details * raise exceptions visible to the frontend * sync plugins on load * access control to updating plugins from the web * access control * remove posthog.json from git * test config schema from json * if you can install via the web, you can also configure * remove separate view access * title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository * add self.team to plugin base class * add instance_init method * refactor into files * sync global plugin config from json * make global plugins work, add test * global plugins in interface, make them take precedence over local plugins * add comments to plugin base class * reload/reset plugins before each test * add error field to plugins * add many plugin zips * add many plugin zips, fix imports * store errors on plugin object and test them * fix types * add null to error * can be with any team ID in the test * save problems running plugins in the plugin_config model * try to create redis connection pool only once * throw if no redis * mypy * get instance inside heartbeat and not top level * try caching pubsub * try pip install with -q * install pip externally * remove uuid and typing, now in stdlib * more verbosity * add pip back * catch exceptions * quiet and no input for pip * check plugin reload every 10sec on new task * fix type errors * fix requirements error message * use repository.json * only load and reload plugins on workers * rename task * support local js plugins via py-mini-racer * load js plugins from zip files * extract jsplugin class and convert to syntax that uses global functions instead of initializing a class * process events via grpc * process events with the "posthog-plugins" queue to enable plugins * remove old native python & mini racer plugin code * default to false * change env vars * fix test * remove grpc tools * skip plugins in migrate.py scripts * fix migration * change output of settings debug banner to STDERR * start posthog plugin server with worker * try to fix python 3.7 test * add fallback for the optional argument * annoying CI test debug * try to finally fix python 3.7 test * here we go again * move plugins under instance * move plugins npm start into its own folder * more console.log debugginf * and again * move plugins to separate script * more prints * fix test error * docker config * small fixes * dckerfile fix * reload plugins via pubsub, upgrade version * plugins that support team setup code * sync if made changes from config * move plugins in menu * require node 14 in heroku for better plugin support (namely ?. support) * bump node version in dockerfiles * update node versions for github actions * update the concurrency for heroku workers * update the concurrency for heroku workers (add link) * Fix migrations after merge * add ignore_result to process_event tasks * fix: docker-preview run in parallel bug * change order of commands * remove separate plugins server conf script * clarify intent * revert castaway change * add context to plugins/sync.py * change everything to ValidationError * delegate destroy to super * no request to repository url if can't install * make the if cleaner * add clarifying line * add clarifying line * fix url field type * rename get_redis_instance to get_client, move to posthog.redis * remove duplicate validation * flip if around * simplify api logic * simplify plugin_config api, fix global_plugin error * remove unnecessary field rename * mypy * Plugins UI (#2090) * base UI * more UI * load plugin image if available * toggle enabled plugin * plugin cards for available * custom plugins * change plugin configuration to drawer * asks for confirmation when enabling or disabling a plugin * loading state * separation of concerns, leave new styles for separate PR * general improvements * remove button when installation is not available * preemptively avoid merge conflict with #2114 * move papercups widget & hide bottom bar when drawer is open * allow clicking the entire plugin card * address all feedback * move plugins under "project" menu * Hide "configure" from globally enabled plugins Co-authored-by: Marius Andra <marius.andra@gmail.com> * add plugins opt in toggle to project/plugins * choose pipeline based on team setting * add "beta" * plugin opt-in opt-out pages * adjust install button * remove tasks that are never called, remove PLUGINS_ENABLED global key * fix responsive card display * fix typo and drawer width * skeleton fixes * typo * use "posthog-plugin-server" npm package * "posthog-plugin-server" doc * require the plugin server to be online before enabling plugins * remove a few needeless "?." cases * add hint for config_schema * add hint for errors * show plugin errors * stop clicks if clicking on error * show plugin errors * loading indicators * reload plugins when opting in/out * nicer beta tag * add frontend type * fix mypy error * fix test * disable plugins if MULTI_TENANCY * upgrade plugin-server version * save event with plugin error * upgrade plugin server * squashed & optimized migrations * remove unused import * updates opt-in copy & hides tech details for cloud version * fix cypress tests * compare with None * change plugins url and add redirect * remove ellipsis * use code snippet in plugin errors * change github regex * fix loading flickering on installing plugins * add comment to plugin archive * fix python style * remove pip-tools (relic from the python plugin era) * hard pin plugin server version * remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway) * update lockfile Co-authored-by: James Greenhill <fuziontech@gmail.com> Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
2020-11-02 15:08:30 +01:00
# this kills all processes when the last one terminates
trap 'kill $(jobs -p)' EXIT
feat(celery): Prepare to run on multiple queues (#19157) * Add Celery queues env file with default queues Reasoning: We need to configure Celery workers in several places to consume from a specific set of queues. * Define some queues
2024-01-17 12:54:12 +01:00
source ./bin/celery-queues.env
Make workers optional for now, add a warning if they are offline (#494) * add warning script to settings and docker-worker in case REDIS_URL is not found * test celery heartbeat * save celery heartbeat in redis * /_stats/ with worker heartbeat * refactor top content * show worker stats * heartbeat delay to 90sec * update worker error message * remove line * add redbeat schdeduler * add redbeat for local development
2020-04-08 14:52:29 +02:00
# start celery worker with heartbeat (-B)
feat: move query performance polling to its own celery task in a performant manner (#22497)
2024-05-31 07:25:08 +02:00
SKIP_ASYNC_MIGRATIONS_SETUP=0 CELERY_WORKER_QUEUES=$CELERY_WORKER_QUEUES celery -A posthog worker --without-heartbeat --without-mingle --pool=threads -Ofair -n node@%h &
celery -A posthog beat -S redbeat.RedBeatScheduler &
Plugin v8 (#1946) * plugin progress * blah * add posthog config for plugins * test gitignore * new functionality for plugins * support local plugin paths * also ignore symlinks * add positional argument * fixes * small fixes * config polish * config passed to posthog plugin * ooooooops * symlink fix * cleanse dir before loading * add cache to plugins * pickle the goods * unlink symlink * pass full config * unlink even if link points to nothing * fix fix * return none if value is empty * plugin model * plugins scene * add config schema to plugins * install plugins * save descriptions * show descriptions * edit plugin * save plugin config * plugin modal * uninstall plugins * UX cleanup * add "required" to plugin config * open plugin modal after install * split to subcomponents * install custom plugins * rework backend for model plugins * Plugins on models * simple reload pubsub * fix apps not installed * fix master/main issue * fix reload command * use the github api to get the default branch * init plugins only if not running migrate/makemigrations * store plugins zip archives in postgres * tag plugins to specific versions * save plugins in pluginConfig * update pluginConfigs instead of adding new rows, remove from redux on uninstall * remove debug * run plugins from db by team * reload when deleting * remove debug * smarter handling of dynamic plugins, support local plugins again * improve typings, add some nicer warnings * yarn lock file after merge * squash migrations and add "locked" field to plugins * error if folder not found in zip * unregister plugins * skip plugin init in test mode * basic plugin test * avoid mutating the same prop hash * add pip tools to requirements.txt * fix mypy, fix manage.py script error * avoid plugins with mypy * mypy fix * abstract redis into plugin and add team_id to reload * refactor and start work on syncing with posthog.json * start testing plugin loading from json * test plugin deletion * test for syncing plugins from config * complete and then test local json plugin sync * test converting back and forward between an local and http path * remove global plugin config from plugins array in posthog.json * rename configSchema --> config_schema * fix migration after merge * rename from_cli to from_json * mypy * import pip after plugin loaded * show error details * raise exceptions visible to the frontend * sync plugins on load * access control to updating plugins from the web * access control * remove posthog.json from git * test config schema from json * if you can install via the web, you can also configure * remove separate view access * title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository * add self.team to plugin base class * add instance_init method * refactor into files * sync global plugin config from json * make global plugins work, add test * global plugins in interface, make them take precedence over local plugins * add comments to plugin base class * reload/reset plugins before each test * add error field to plugins * add many plugin zips * add many plugin zips, fix imports * store errors on plugin object and test them * fix types * add null to error * can be with any team ID in the test * save problems running plugins in the plugin_config model * try to create redis connection pool only once * throw if no redis * mypy * get instance inside heartbeat and not top level * try caching pubsub * try pip install with -q * install pip externally * remove uuid and typing, now in stdlib * more verbosity * add pip back * catch exceptions * quiet and no input for pip * check plugin reload every 10sec on new task * fix type errors * fix requirements error message * use repository.json * only load and reload plugins on workers * rename task * support local js plugins via py-mini-racer * load js plugins from zip files * extract jsplugin class and convert to syntax that uses global functions instead of initializing a class * process events via grpc * process events with the "posthog-plugins" queue to enable plugins * remove old native python & mini racer plugin code * default to false * change env vars * fix test * remove grpc tools * skip plugins in migrate.py scripts * fix migration * change output of settings debug banner to STDERR * start posthog plugin server with worker * try to fix python 3.7 test * add fallback for the optional argument * annoying CI test debug * try to finally fix python 3.7 test * here we go again * move plugins under instance * move plugins npm start into its own folder * more console.log debugginf * and again * move plugins to separate script * more prints * fix test error * docker config * small fixes * dckerfile fix * reload plugins via pubsub, upgrade version * plugins that support team setup code * sync if made changes from config * move plugins in menu * require node 14 in heroku for better plugin support (namely ?. support) * bump node version in dockerfiles * update node versions for github actions * update the concurrency for heroku workers * update the concurrency for heroku workers (add link) * Fix migrations after merge * add ignore_result to process_event tasks * fix: docker-preview run in parallel bug * change order of commands * remove separate plugins server conf script * clarify intent * revert castaway change * add context to plugins/sync.py * change everything to ValidationError * delegate destroy to super * no request to repository url if can't install * make the if cleaner * add clarifying line * add clarifying line * fix url field type * rename get_redis_instance to get_client, move to posthog.redis * remove duplicate validation * flip if around * simplify api logic * simplify plugin_config api, fix global_plugin error * remove unnecessary field rename * mypy * Plugins UI (#2090) * base UI * more UI * load plugin image if available * toggle enabled plugin * plugin cards for available * custom plugins * change plugin configuration to drawer * asks for confirmation when enabling or disabling a plugin * loading state * separation of concerns, leave new styles for separate PR * general improvements * remove button when installation is not available * preemptively avoid merge conflict with #2114 * move papercups widget & hide bottom bar when drawer is open * allow clicking the entire plugin card * address all feedback * move plugins under "project" menu * Hide "configure" from globally enabled plugins Co-authored-by: Marius Andra <marius.andra@gmail.com> * add plugins opt in toggle to project/plugins * choose pipeline based on team setting * add "beta" * plugin opt-in opt-out pages * adjust install button * remove tasks that are never called, remove PLUGINS_ENABLED global key * fix responsive card display * fix typo and drawer width * skeleton fixes * typo * use "posthog-plugin-server" npm package * "posthog-plugin-server" doc * require the plugin server to be online before enabling plugins * remove a few needeless "?." cases * add hint for config_schema * add hint for errors * show plugin errors * stop clicks if clicking on error * show plugin errors * loading indicators * reload plugins when opting in/out * nicer beta tag * add frontend type * fix mypy error * fix test * disable plugins if MULTI_TENANCY * upgrade plugin-server version * save event with plugin error * upgrade plugin server * squashed & optimized migrations * remove unused import * updates opt-in copy & hides tech details for cloud version * fix cypress tests * compare with None * change plugins url and add redirect * remove ellipsis * use code snippet in plugin errors * change github regex * fix loading flickering on installing plugins * add comment to plugin archive * fix python style * remove pip-tools (relic from the python plugin era) * hard pin plugin server version * remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway) * update lockfile Co-authored-by: James Greenhill <fuziontech@gmail.com> Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
2020-11-02 15:08:30 +01:00
refactor(plugin-server): Remove --idle mode (#9798) * refactor(plugin-server): Remove `--idle` mode * Support `PLUGIN_SERVER_IDLE` in `bin/start-worker` * Don't `export` needlessly in npm script
2022-05-16 18:22:49 +02:00
if [[ "$PLUGIN_SERVER_IDLE" != "1" && "$PLUGIN_SERVER_IDLE" != "true" ]]; then
./bin/plugin-server
fi
feat: Add Temporal to the dev and hobby stacks (#14658) * feat: Add Temporal to the dev and hobby stacks * disable elastic for hobby because of resources * checkpoint * update requirements * worker is up, but without the sandbox * ensure temporal does not depend on elastic * Feedbacked * pip-compile dev * mypy fixes * add a bit of colorful logging * add django temporal worker to the mix * checkpoint for dev-full docker * Working on docker-full, but checkpointing for now * add migration bits for full
2023-03-20 23:56:14 +01:00
feat: Celery tasks refactor (#19827)
2024-01-18 14:28:10 +01:00
wait
Copy Permalink