posthog/bin/start-worker

#!/bin/bash
set -e

# this kills all processes when the last one terminates
trap 'kill $(jobs -p)' EXIT

# start celery worker with heartbeat (-B)
celery -A posthog worker -B --scheduler redbeat.RedBeatScheduler --without-heartbeat --without-gossip --without-mingle -Ofair &

# start celery plugin worker
./bin/plugin-server
scripts to run background, worker and frontend simultaneously 2020-04-07 12:34:01 +02:00			`#!/bin/bash`
			`set -e`

Plugin v8 (#1946) * plugin progress * blah * add posthog config for plugins * test gitignore * new functionality for plugins * support local plugin paths * also ignore symlinks * add positional argument * fixes * small fixes * config polish * config passed to posthog plugin * ooooooops * symlink fix * cleanse dir before loading * add cache to plugins * pickle the goods * unlink symlink * pass full config * unlink even if link points to nothing * fix fix * return none if value is empty * plugin model * plugins scene * add config schema to plugins * install plugins * save descriptions * show descriptions * edit plugin * save plugin config * plugin modal * uninstall plugins * UX cleanup * add "required" to plugin config * open plugin modal after install * split to subcomponents * install custom plugins * rework backend for model plugins * Plugins on models * simple reload pubsub * fix apps not installed * fix master/main issue * fix reload command * use the github api to get the default branch * init plugins only if not running migrate/makemigrations * store plugins zip archives in postgres * tag plugins to specific versions * save plugins in pluginConfig * update pluginConfigs instead of adding new rows, remove from redux on uninstall * remove debug * run plugins from db by team * reload when deleting * remove debug * smarter handling of dynamic plugins, support local plugins again * improve typings, add some nicer warnings * yarn lock file after merge * squash migrations and add "locked" field to plugins * error if folder not found in zip * unregister plugins * skip plugin init in test mode * basic plugin test * avoid mutating the same prop hash * add pip tools to requirements.txt * fix mypy, fix manage.py script error * avoid plugins with mypy * mypy fix * abstract redis into plugin and add team_id to reload * refactor and start work on syncing with posthog.json * start testing plugin loading from json * test plugin deletion * test for syncing plugins from config * complete and then test local json plugin sync * test converting back and forward between an local and http path * remove global plugin config from plugins array in posthog.json * rename configSchema --> config_schema * fix migration after merge * rename from_cli to from_json * mypy * import pip after plugin loaded * show error details * raise exceptions visible to the frontend * sync plugins on load * access control to updating plugins from the web * access control * remove posthog.json from git * test config schema from json * if you can install via the web, you can also configure * remove separate view access * title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository * add self.team to plugin base class * add instance_init method * refactor into files * sync global plugin config from json * make global plugins work, add test * global plugins in interface, make them take precedence over local plugins * add comments to plugin base class * reload/reset plugins before each test * add error field to plugins * add many plugin zips * add many plugin zips, fix imports * store errors on plugin object and test them * fix types * add null to error * can be with any team ID in the test * save problems running plugins in the plugin_config model * try to create redis connection pool only once * throw if no redis * mypy * get instance inside heartbeat and not top level * try caching pubsub * try pip install with -q * install pip externally * remove uuid and typing, now in stdlib * more verbosity * add pip back * catch exceptions * quiet and no input for pip * check plugin reload every 10sec on new task * fix type errors * fix requirements error message * use repository.json * only load and reload plugins on workers * rename task * support local js plugins via py-mini-racer * load js plugins from zip files * extract jsplugin class and convert to syntax that uses global functions instead of initializing a class * process events via grpc * process events with the "posthog-plugins" queue to enable plugins * remove old native python & mini racer plugin code * default to false * change env vars * fix test * remove grpc tools * skip plugins in migrate.py scripts * fix migration * change output of settings debug banner to STDERR * start posthog plugin server with worker * try to fix python 3.7 test * add fallback for the optional argument * annoying CI test debug * try to finally fix python 3.7 test * here we go again * move plugins under instance * move plugins npm start into its own folder * more console.log debugginf * and again * move plugins to separate script * more prints * fix test error * docker config * small fixes * dckerfile fix * reload plugins via pubsub, upgrade version * plugins that support team setup code * sync if made changes from config * move plugins in menu * require node 14 in heroku for better plugin support (namely ?. support) * bump node version in dockerfiles * update node versions for github actions * update the concurrency for heroku workers * update the concurrency for heroku workers (add link) * Fix migrations after merge * add ignore_result to process_event tasks * fix: docker-preview run in parallel bug * change order of commands * remove separate plugins server conf script * clarify intent * revert castaway change * add context to plugins/sync.py * change everything to ValidationError * delegate destroy to super * no request to repository url if can't install * make the if cleaner * add clarifying line * add clarifying line * fix url field type * rename get_redis_instance to get_client, move to posthog.redis * remove duplicate validation * flip if around * simplify api logic * simplify plugin_config api, fix global_plugin error * remove unnecessary field rename * mypy * Plugins UI (#2090) * base UI * more UI * load plugin image if available * toggle enabled plugin * plugin cards for available * custom plugins * change plugin configuration to drawer * asks for confirmation when enabling or disabling a plugin * loading state * separation of concerns, leave new styles for separate PR * general improvements * remove button when installation is not available * preemptively avoid merge conflict with #2114 * move papercups widget & hide bottom bar when drawer is open * allow clicking the entire plugin card * address all feedback * move plugins under "project" menu * Hide "configure" from globally enabled plugins Co-authored-by: Marius Andra <marius.andra@gmail.com> * add plugins opt in toggle to project/plugins * choose pipeline based on team setting * add "beta" * plugin opt-in opt-out pages * adjust install button * remove tasks that are never called, remove PLUGINS_ENABLED global key * fix responsive card display * fix typo and drawer width * skeleton fixes * typo * use "posthog-plugin-server" npm package * "posthog-plugin-server" doc * require the plugin server to be online before enabling plugins * remove a few needeless "?." cases * add hint for config_schema * add hint for errors * show plugin errors * stop clicks if clicking on error * show plugin errors * loading indicators * reload plugins when opting in/out * nicer beta tag * add frontend type * fix mypy error * fix test * disable plugins if MULTI_TENANCY * upgrade plugin-server version * save event with plugin error * upgrade plugin server * squashed & optimized migrations * remove unused import * updates opt-in copy & hides tech details for cloud version * fix cypress tests * compare with None * change plugins url and add redirect * remove ellipsis * use code snippet in plugin errors * change github regex * fix loading flickering on installing plugins * add comment to plugin archive * fix python style * remove pip-tools (relic from the python plugin era) * hard pin plugin server version * remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway) * update lockfile Co-authored-by: James Greenhill <fuziontech@gmail.com> Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com> 2020-11-02 15:08:30 +01:00			`# this kills all processes when the last one terminates`
			`trap 'kill $(jobs -p)' EXIT`

Make workers optional for now, add a warning if they are offline (#494) * add warning script to settings and docker-worker in case REDIS_URL is not found * test celery heartbeat * save celery heartbeat in redis * /_stats/ with worker heartbeat * refactor top content * show worker stats * heartbeat delay to 90sec * update worker error message * remove line * add redbeat schdeduler * add redbeat for local development 2020-04-08 14:52:29 +02:00			`# start celery worker with heartbeat (-B)`
Use -Ofair for celery workers (#5063) From https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb > By default, preforking Celery workers distribute tasks to their worker processes as soon as they are received, regardless of whether the process is currently busy with other tasks. > If you have a set of tasks that take varying amounts of time to complete — either deliberately or due to unpredictable network conditions, etc. — this will cause unexpected delays in total execution time for tasks in the queue. This is 100% the case for us. This should "load balance" the tasks better across workers. 2021-07-12 12:52:05 +02:00			`celery -A posthog worker -B --scheduler redbeat.RedBeatScheduler --without-heartbeat --without-gossip --without-mingle -Ofair &`
Plugin v8 (#1946) * plugin progress * blah * add posthog config for plugins * test gitignore * new functionality for plugins * support local plugin paths * also ignore symlinks * add positional argument * fixes * small fixes * config polish * config passed to posthog plugin * ooooooops * symlink fix * cleanse dir before loading * add cache to plugins * pickle the goods * unlink symlink * pass full config * unlink even if link points to nothing * fix fix * return none if value is empty * plugin model * plugins scene * add config schema to plugins * install plugins * save descriptions * show descriptions * edit plugin * save plugin config * plugin modal * uninstall plugins * UX cleanup * add "required" to plugin config * open plugin modal after install * split to subcomponents * install custom plugins * rework backend for model plugins * Plugins on models * simple reload pubsub * fix apps not installed * fix master/main issue * fix reload command * use the github api to get the default branch * init plugins only if not running migrate/makemigrations * store plugins zip archives in postgres * tag plugins to specific versions * save plugins in pluginConfig * update pluginConfigs instead of adding new rows, remove from redux on uninstall * remove debug * run plugins from db by team * reload when deleting * remove debug * smarter handling of dynamic plugins, support local plugins again * improve typings, add some nicer warnings * yarn lock file after merge * squash migrations and add "locked" field to plugins * error if folder not found in zip * unregister plugins * skip plugin init in test mode * basic plugin test * avoid mutating the same prop hash * add pip tools to requirements.txt * fix mypy, fix manage.py script error * avoid plugins with mypy * mypy fix * abstract redis into plugin and add team_id to reload * refactor and start work on syncing with posthog.json * start testing plugin loading from json * test plugin deletion * test for syncing plugins from config * complete and then test local json plugin sync * test converting back and forward between an local and http path * remove global plugin config from plugins array in posthog.json * rename configSchema --> config_schema * fix migration after merge * rename from_cli to from_json * mypy * import pip after plugin loaded * show error details * raise exceptions visible to the frontend * sync plugins on load * access control to updating plugins from the web * access control * remove posthog.json from git * test config schema from json * if you can install via the web, you can also configure * remove separate view access * title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository * add self.team to plugin base class * add instance_init method * refactor into files * sync global plugin config from json * make global plugins work, add test * global plugins in interface, make them take precedence over local plugins * add comments to plugin base class * reload/reset plugins before each test * add error field to plugins * add many plugin zips * add many plugin zips, fix imports * store errors on plugin object and test them * fix types * add null to error * can be with any team ID in the test * save problems running plugins in the plugin_config model * try to create redis connection pool only once * throw if no redis * mypy * get instance inside heartbeat and not top level * try caching pubsub * try pip install with -q * install pip externally * remove uuid and typing, now in stdlib * more verbosity * add pip back * catch exceptions * quiet and no input for pip * check plugin reload every 10sec on new task * fix type errors * fix requirements error message * use repository.json * only load and reload plugins on workers * rename task * support local js plugins via py-mini-racer * load js plugins from zip files * extract jsplugin class and convert to syntax that uses global functions instead of initializing a class * process events via grpc * process events with the "posthog-plugins" queue to enable plugins * remove old native python & mini racer plugin code * default to false * change env vars * fix test * remove grpc tools * skip plugins in migrate.py scripts * fix migration * change output of settings debug banner to STDERR * start posthog plugin server with worker * try to fix python 3.7 test * add fallback for the optional argument * annoying CI test debug * try to finally fix python 3.7 test * here we go again * move plugins under instance * move plugins npm start into its own folder * more console.log debugginf * and again * move plugins to separate script * more prints * fix test error * docker config * small fixes * dckerfile fix * reload plugins via pubsub, upgrade version * plugins that support team setup code * sync if made changes from config * move plugins in menu * require node 14 in heroku for better plugin support (namely ?. support) * bump node version in dockerfiles * update node versions for github actions * update the concurrency for heroku workers * update the concurrency for heroku workers (add link) * Fix migrations after merge * add ignore_result to process_event tasks * fix: docker-preview run in parallel bug * change order of commands * remove separate plugins server conf script * clarify intent * revert castaway change * add context to plugins/sync.py * change everything to ValidationError * delegate destroy to super * no request to repository url if can't install * make the if cleaner * add clarifying line * add clarifying line * fix url field type * rename get_redis_instance to get_client, move to posthog.redis * remove duplicate validation * flip if around * simplify api logic * simplify plugin_config api, fix global_plugin error * remove unnecessary field rename * mypy * Plugins UI (#2090) * base UI * more UI * load plugin image if available * toggle enabled plugin * plugin cards for available * custom plugins * change plugin configuration to drawer * asks for confirmation when enabling or disabling a plugin * loading state * separation of concerns, leave new styles for separate PR * general improvements * remove button when installation is not available * preemptively avoid merge conflict with #2114 * move papercups widget & hide bottom bar when drawer is open * allow clicking the entire plugin card * address all feedback * move plugins under "project" menu * Hide "configure" from globally enabled plugins Co-authored-by: Marius Andra <marius.andra@gmail.com> * add plugins opt in toggle to project/plugins * choose pipeline based on team setting * add "beta" * plugin opt-in opt-out pages * adjust install button * remove tasks that are never called, remove PLUGINS_ENABLED global key * fix responsive card display * fix typo and drawer width * skeleton fixes * typo * use "posthog-plugin-server" npm package * "posthog-plugin-server" doc * require the plugin server to be online before enabling plugins * remove a few needeless "?." cases * add hint for config_schema * add hint for errors * show plugin errors * stop clicks if clicking on error * show plugin errors * loading indicators * reload plugins when opting in/out * nicer beta tag * add frontend type * fix mypy error * fix test * disable plugins if MULTI_TENANCY * upgrade plugin-server version * save event with plugin error * upgrade plugin server * squashed & optimized migrations * remove unused import * updates opt-in copy & hides tech details for cloud version * fix cypress tests * compare with None * change plugins url and add redirect * remove ellipsis * use code snippet in plugin errors * change github regex * fix loading flickering on installing plugins * add comment to plugin archive * fix python style * remove pip-tools (relic from the python plugin era) * hard pin plugin server version * remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway) * update lockfile Co-authored-by: James Greenhill <fuziontech@gmail.com> Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com> 2020-11-02 15:08:30 +01:00
			`# start celery plugin worker`
			`./bin/plugin-server`