Plugin v8 (#1946)
* plugin progress
* blah
* add posthog config for plugins
* test gitignore
* new functionality for plugins
* support local plugin paths
* also ignore symlinks
* add positional argument
* fixes
* small fixes
* config polish
* config passed to posthog plugin
* ooooooops
* symlink fix
* cleanse dir before loading
* add cache to plugins
* pickle the goods
* unlink symlink
* pass full config
* unlink even if link points to nothing
* fix fix
* return none if value is empty
* plugin model
* plugins scene
* add config schema to plugins
* install plugins
* save descriptions
* show descriptions
* edit plugin
* save plugin config
* plugin modal
* uninstall plugins
* UX cleanup
* add "required" to plugin config
* open plugin modal after install
* split to subcomponents
* install custom plugins
* rework backend for model plugins
* Plugins on models
* simple reload pubsub
* fix apps not installed
* fix master/main issue
* fix reload command
* use the github api to get the default branch
* init plugins only if not running migrate/makemigrations
* store plugins zip archives in postgres
* tag plugins to specific versions
* save plugins in pluginConfig
* update pluginConfigs instead of adding new rows, remove from redux on uninstall
* remove debug
* run plugins from db by team
* reload when deleting
* remove debug
* smarter handling of dynamic plugins, support local plugins again
* improve typings, add some nicer warnings
* yarn lock file after merge
* squash migrations and add "locked" field to plugins
* error if folder not found in zip
* unregister plugins
* skip plugin init in test mode
* basic plugin test
* avoid mutating the same prop hash
* add pip tools to requirements.txt
* fix mypy, fix manage.py script error
* avoid plugins with mypy
* mypy fix
* abstract redis into plugin and add team_id to reload
* refactor and start work on syncing with posthog.json
* start testing plugin loading from json
* test plugin deletion
* test for syncing plugins from config
* complete and then test local json plugin sync
* test converting back and forward between an local and http path
* remove global plugin config from plugins array in posthog.json
* rename configSchema --> config_schema
* fix migration after merge
* rename from_cli to from_json
* mypy
* import pip after plugin loaded
* show error details
* raise exceptions visible to the frontend
* sync plugins on load
* access control to updating plugins from the web
* access control
* remove posthog.json from git
* test config schema from json
* if you can install via the web, you can also configure
* remove separate view access
* title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository
* add self.team to plugin base class
* add instance_init method
* refactor into files
* sync global plugin config from json
* make global plugins work, add test
* global plugins in interface, make them take precedence over local plugins
* add comments to plugin base class
* reload/reset plugins before each test
* add error field to plugins
* add many plugin zips
* add many plugin zips, fix imports
* store errors on plugin object and test them
* fix types
* add null to error
* can be with any team ID in the test
* save problems running plugins in the plugin_config model
* try to create redis connection pool only once
* throw if no redis
* mypy
* get instance inside heartbeat and not top level
* try caching pubsub
* try pip install with -q
* install pip externally
* remove uuid and typing, now in stdlib
* more verbosity
* add pip back
* catch exceptions
* quiet and no input for pip
* check plugin reload every 10sec on new task
* fix type errors
* fix requirements error message
* use repository.json
* only load and reload plugins on workers
* rename task
* support local js plugins via py-mini-racer
* load js plugins from zip files
* extract jsplugin class and convert to syntax that uses global functions instead of initializing a class
* process events via grpc
* process events with the "posthog-plugins" queue to enable plugins
* remove old native python & mini racer plugin code
* default to false
* change env vars
* fix test
* remove grpc tools
* skip plugins in migrate.py scripts
* fix migration
* change output of settings debug banner to STDERR
* start posthog plugin server with worker
* try to fix python 3.7 test
* add fallback for the optional argument
* annoying CI test debug
* try to finally fix python 3.7 test
* here we go again
* move plugins under instance
* move plugins npm start into its own folder
* more console.log debugginf
* and again
* move plugins to separate script
* more prints
* fix test error
* docker config
* small fixes
* dckerfile fix
* reload plugins via pubsub, upgrade version
* plugins that support team setup code
* sync if made changes from config
* move plugins in menu
* require node 14 in heroku for better plugin support (namely ?. support)
* bump node version in dockerfiles
* update node versions for github actions
* update the concurrency for heroku workers
* update the concurrency for heroku workers (add link)
* Fix migrations after merge
* add ignore_result to process_event tasks
* fix: docker-preview run in parallel bug
* change order of commands
* remove separate plugins server conf script
* clarify intent
* revert castaway change
* add context to plugins/sync.py
* change everything to ValidationError
* delegate destroy to super
* no request to repository url if can't install
* make the if cleaner
* add clarifying line
* add clarifying line
* fix url field type
* rename get_redis_instance to get_client, move to posthog.redis
* remove duplicate validation
* flip if around
* simplify api logic
* simplify plugin_config api, fix global_plugin error
* remove unnecessary field rename
* mypy
* Plugins UI (#2090)
* base UI
* more UI
* load plugin image if available
* toggle enabled plugin
* plugin cards for available
* custom plugins
* change plugin configuration to drawer
* asks for confirmation when enabling or disabling a plugin
* loading state
* separation of concerns, leave new styles for separate PR
* general improvements
* remove button when installation is not available
* preemptively avoid merge conflict with #2114
* move papercups widget & hide bottom bar when drawer is open
* allow clicking the entire plugin card
* address all feedback
* move plugins under "project" menu
* Hide "configure" from globally enabled plugins
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add plugins opt in toggle to project/plugins
* choose pipeline based on team setting
* add "beta"
* plugin opt-in opt-out pages
* adjust install button
* remove tasks that are never called, remove PLUGINS_ENABLED global key
* fix responsive card display
* fix typo and drawer width
* skeleton fixes
* typo
* use "posthog-plugin-server" npm package
* "posthog-plugin-server" doc
* require the plugin server to be online before enabling plugins
* remove a few needeless "?." cases
* add hint for config_schema
* add hint for errors
* show plugin errors
* stop clicks if clicking on error
* show plugin errors
* loading indicators
* reload plugins when opting in/out
* nicer beta tag
* add frontend type
* fix mypy error
* fix test
* disable plugins if MULTI_TENANCY
* upgrade plugin-server version
* save event with plugin error
* upgrade plugin server
* squashed & optimized migrations
* remove unused import
* updates opt-in copy & hides tech details for cloud version
* fix cypress tests
* compare with None
* change plugins url and add redirect
* remove ellipsis
* use code snippet in plugin errors
* change github regex
* fix loading flickering on installing plugins
* add comment to plugin archive
* fix python style
* remove pip-tools (relic from the python plugin era)
* hard pin plugin server version
* remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway)
* update lockfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
2020-11-02 15:08:30 +01:00
|
|
|
#!/bin/bash
|
2020-12-14 16:05:18 +01:00
|
|
|
|
2021-01-21 15:39:44 +01:00
|
|
|
while test $# -gt 0; do
|
|
|
|
case "$1" in
|
|
|
|
-h|--help)
|
|
|
|
echo "USAGE:"
|
|
|
|
echo " bin/plugin-server [FLAGS]"
|
|
|
|
echo " "
|
|
|
|
echo "FLAGS:"
|
|
|
|
echo " -h, --help Print this help information."
|
|
|
|
echo " --no-restart-loop Run without restart loop. Recommended when deferring resiliency to e.g. docker-compose."
|
|
|
|
exit 0
|
|
|
|
;;
|
|
|
|
--no-restart-loop)
|
|
|
|
NO_RESTART_LOOP='true'
|
|
|
|
shift
|
|
|
|
;;
|
|
|
|
*)
|
|
|
|
break
|
|
|
|
;;
|
|
|
|
esac
|
|
|
|
done
|
|
|
|
|
|
|
|
if [[ -n $POSTHOG_DB_NAME ]]; then
|
2021-02-22 19:15:45 +01:00
|
|
|
[[ -n $POSTHOG_DB_PASSWORD ]] && DB_PASSWORD_COMPONENT=":$POSTHOG_DB_PASSWORD"
|
|
|
|
export DATABASE_URL="postgres://${POSTHOG_DB_USER:-"postgres"}${DB_PASSWORD_COMPONENT}@${POSTHOG_POSTGRES_HOST:-"localhost"}:${POSTHOG_POSTGRES_PORT:-"5432"}/${POSTHOG_DB_NAME}"
|
|
|
|
fi
|
|
|
|
|
|
|
|
if [[ -z $REDIS_URL ]]; then
|
2021-02-23 13:04:33 +01:00
|
|
|
[[ -n $POSTHOG_REDIS_PASSWORD ]] && REDIS_PASSWORD_COMPONENT=":$POSTHOG_REDIS_PASSWORD@"
|
2021-02-22 19:15:45 +01:00
|
|
|
export REDIS_URL="redis://${REDIS_PASSWORD_COMPONENT}${POSTHOG_REDIS_HOST:-"localhost"}:${POSTHOG_REDIS_PORT:-"6379"}/"
|
2021-01-21 15:39:44 +01:00
|
|
|
fi
|
2020-12-09 18:08:25 +01:00
|
|
|
|
2021-01-21 15:39:44 +01:00
|
|
|
export BASE_DIR=$(dirname $(dirname "$PWD/${0#./}"))
|
|
|
|
# Crudely extract netlocs by removing "kafka://" in a compatibility approach
|
|
|
|
export KAFKA_HOSTS=${KAFKA_HOSTS:-$(echo $KAFKA_URL | sed -e "s/kafka\(\+ssl\)\{0,1\}:\/\///g")}
|
|
|
|
# Check PRIMARY_DB in a compatibility approach
|
|
|
|
export KAFKA_ENABLED=${KAFKA_ENABLED:-$([[ $PRIMARY_DB = "clickhouse" ]] && echo "true" || echo "false")}
|
2020-12-09 18:08:25 +01:00
|
|
|
|
|
|
|
# On _Heroku_, the $WEB_CONCURRENCY env contains suggested number of workers per dyno.
|
|
|
|
# Unfortunately we are running a NodeJS app, yet get the value for the "python" buildpack.
|
|
|
|
# Thus instead of using this env directly, calculate the real concurrency from $DYNO_RAM.
|
|
|
|
# Python: https://github.com/heroku/heroku-buildpack-python/blob/main/vendor/WEB_CONCURRENCY.sh
|
|
|
|
# NodeJS: https://devcenter.heroku.com/articles/node-concurrency#common-runtime
|
2021-01-21 15:39:44 +01:00
|
|
|
if [[ -n $DYNO_RAM ]]; then
|
2020-12-09 18:08:25 +01:00
|
|
|
# One worker for 512MB of RAM, rounding up
|
|
|
|
export WORKER_CONCURRENCY=$(( ($DYNO_RAM - 1) / 512 + 1 ))
|
|
|
|
fi
|
|
|
|
|
2021-02-23 10:26:26 +01:00
|
|
|
# Help debug redis connection issues
|
|
|
|
export DEBUG=ioredis:*
|
|
|
|
|
2020-12-10 07:29:06 +01:00
|
|
|
cd plugins
|
|
|
|
|
2021-01-21 15:39:44 +01:00
|
|
|
echo "🧐 Verifying installed packages..."
|
|
|
|
yarn --frozen-lockfile
|
2020-12-10 07:29:06 +01:00
|
|
|
if [ $? -ne 0 ]; then
|
2021-01-21 15:39:44 +01:00
|
|
|
echo "💥 Verification failed!"
|
|
|
|
exit 1
|
2020-12-10 07:29:06 +01:00
|
|
|
fi
|
|
|
|
|
2021-01-21 15:39:44 +01:00
|
|
|
if [[ -n $NO_RESTART_LOOP ]]; then
|
|
|
|
echo "▶️ Starting plugin server..."
|
2020-12-17 02:14:57 +01:00
|
|
|
yarn start
|
2021-01-21 15:39:44 +01:00
|
|
|
else
|
|
|
|
echo "🔁 Starting plugin server in a resiliency loop..."
|
|
|
|
while true; do
|
|
|
|
yarn start
|
|
|
|
echo "💥 Plugin server crashed!"
|
|
|
|
echo "⌛️ Waiting 2 seconds before restarting..."
|
|
|
|
sleep 2
|
|
|
|
done
|
|
|
|
fi
|