* fix: dont OOM deleting large teams/organizations
Another issue plaguing team deletion was that django ORM loads all
objects for a team into memory. If a team had >200k people, that would
cause the process to fail.
We now instead delete via a single sql statement.
* Change crontab parsing logic
* Cron for deleting clickhouse data
* Delete data conditionally
* Add schedule to tasks
* refactor: Eliminate the `KAFKA_ENABLED` setting
* Remove dead code
* Consolidate plugin server test scripts and CI
* Fix CI command
* Remove Celery queues
* Rearrange test directories
* Update import paths
* Make settings dynamic for MATERIALIZED_COLUMNS_ENABLED
* migrate MATERIALIZED_COLUMNS_ENABLED to this and test
* allow tests that use materialized columns to hit django model for settings
* nicer installed apps config
* update snapshot from test
* make test_property more deterministic
Co-authored-by: James Greenhill <fuzionech@gmail.com>
Co-authored-by: yakkomajuri <yakko.majuri@gmail.com>
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
* Avoid naming collisions when materializing columns
1. Prefix person properties differently. Mixing them up can break
ambigious column issues
2. When name already exists, suffix with random junk :)
* Implement analyze.py
* Add `suggested_columns_to_materialize`
* Add code to backfill a materialized column
* Add tests for backfilling data
* Cleanup
* Add tests for analyze
* WIP: Crontab for materializing columns
* Nooped task for materializing properties
* Use mutations_sync=0 for column tests
* Add comment
* Add migration marking existing columns as materialized with a comment
* Add method to allow looking up materialized columns
* Add test for fetching and creating materialized columns
* Disable caching in tests
* Remove some callsites to CLICKHOUSE_DENORMALIZED_PROPERTIES
* Fixup caching logic
* Add (currently broken) test for denormalizing person properties
* Update funnel event query to work with new functionality
* Update other tests/queries
* Remove now-unused setting
* Enable PLUGIN_SERVER_INGESTION_HANDOFF = get_bool_from_env("PLUGIN_SERVER_INGESTION_HANDOFF
* Don't set PLUGIN_SERVER_INGESTION_HANDOFF in worker
* Add comments
* Remove _HANDOFF from PLUGIN_SERVER_INGESTION
* add stats counter for plugin server handoff, so we can verify events out and events in
* add whitelisted posthog and kea organizations
* disable ingestion this round --> first let's just check the plugin server can talk to kafka & clickhouse before sending real events to it
* enable ingestion in docker-compose.ch.yml
* eliminate bad merge
* async action event matching when using postgres plugin server ingestion (#3182)
* fix org
* remove _HANDOFF from topic
* add plugin_ to plugin server ingestion topic
* update plugin server to 0.7.0
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Fix unix-dgram build
* Add per-organization whitelisting of plugins API on Cloud
* Add migration
* Update can_install_plugins_via_api calls in serializer
* Remove unused type: ignore
* Update PluginSerializer for organization_id
* Make Plugin.organization nullable
* Fix can_install_plugins_via_api
* Update pluginsLogic.ts
* Adjust can_configure_plugins_via_api for organization
* Don't include organization_id in serializer
* Set default Plugin.organization to first Organization in instance
* Fix PluginConfigSerializer.create
* Test that other orgs' plugins can't be accessed (as if they didn't exist, 404)
* fix typo
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Polish system status page
* Show system health badge to is_staff users on Cloud
* Update Cypress test
* Correct inconsistent scene name
* Use RDBMS enum instead of constants
* Add "Analytics database in use" to System Status
* Fix typing
* refactor weekly email task to schedule one async job per team
* optimise async email processing
* turns on weekly email cron job if env var is True
* ensure weekly report emails are sent only once to each email address
* fix mypy
* add test runner to ease pycharm dev
* fix broken import
* drop and recreate the clickhouse test db before running tests
* fix person uuid str json serialization issue
* make kafka optional in tests
* fix inits
* remove need for kafka in person.py
* fix a bunch of mypy errors
* fix function and add process_event to pipeline
* fixed missing params and tests
* change uuid and fix types
* types
* optimize for merge prop test
* make ClickhouseProducer to produce to clickhouse one way or another
* annotate types
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Add missing migration
* Add generate_random_token() model util
* Move PublicTokenAuthentication to utils
* Make use of generate_random_token
* Add User.personal_access_token field
* Add PersonalAccessTokenAuthentication
* Fix PublicTokenAuthentication
* Fix migration and auth import
* Add personal_access_token to user API
* Update Setup.js
* Support trailing slash in API
* Improve PAT auth quality
* Add django-rest-hooks requirement
* Update settings.py for rest_hooks
* Fix django-rest-hooks requirement
* Bring back API routes with no double trailing slash
* Rename posthog.api.team to team_user
* Add API TODO
* Ad PAT auth with X-PAT HTTP header
* Replace User.personal_access_token with PersonalAPIKey model
* Fix PersonalAPIKey max_lengths
* Describe posthog.models.utils.generate_random_token better
* Add personal_api_key to API
* Add authenticate_header to PersonalAPIKeyAuthentication
* Add hook API endpoint
* Use django.utils.timezone in place of datetime.datetime
* Add Personal API Keys to Setup
* Sort personal_api_keys in ORM
* Add Action.on_perform()
* Remove requirements.txt comment
* Add a
* Add REST hook tasks
* Optimize PersonalAPIKeyAuthentication query
* Add a trailing slash version of /e endpoint
* Add team field to PersonalAPIKey model
* Add personal API key support to capture endpoint, get_cached_from_token
* Reject personal API keys from inactive users
* Add extra_properties_json field to /capture
* Improve PAK auth header regex
* Use custom hook model
* Deliver hooks
* Handle action.on_perform
* Consolidate userLogic in userLogic.tsx
* Update PersonalAPIKeys.js
* Make PersonalAPIKey foreign keys read-only
* Update requirements/dev.txt
* Make PersonalAPIKeys TSX
* Fix conflict
* Fix migration
* Fix minor mishaps
* Update and fix tests
* Use CharField of random 32 bits as hook.id
* Fix conflicting migrations
* Fix ValidationError in HookSerializer.validate_event
* Use query param in /api/event/actions ID filtering
* Rename endpoint `hook` to `hooks`
* Satisfy mypy
* Add tests
* Use DRF serialization in action_defined and annotation_created triggers
* Update migration leafs
* Make mypy ignore rest_hooks
* Update Django signal receiver names
* Update TS dependencies
* Revert "Update TS dependencies"
This reverts commit 7fc26fefcd.
* Add field user to Hook model
* Update migration leafs
* Fix circular import
* Fix some code
* Install git before running pip install in Dockerfiles
* Improve personal API keys UI
* Satisfy mypy
* Reword key label placeholder
* Add personal API key support to /api/user/*
Unfortunately these endpoints are still limited by CSRF protections at the moment, so not accessible outside PostHog itself.
* Improve PersonalAPIKeyAuthentication and add CsrfOrKeyViewMiddleware
* Run collectstatic before test
* Don't install dev dependencies in CI
* Update dependency installation order in CI
* Fix bug and describe PersonalAPIKeyAuthentication
* Fix CI issues
* Fix typing issues
* Fix more typing issues
* Use /api/personal_api_keys to list keys
* Move REST hooks (and therefore Zapier) to ee/
* Refactor personal API logic with kea-loaders
* Add "More about API authentication in PostHog docs."
* Update PersonalAPIKeys.tsx
* Use TestMixin
* Fix "Authentication" that should've been "Authorization"
* Add option to skip self.client.force_login in API tests
* Include team_id and user_id in personal API key serialization
* Update test_hooks.py
* Add personal API key tests
* Remove leftover
* Make ee.settings override posthog.settings
* Don't directly import from models
* Remove unused imports
* Fix mypy issues
* Fix HOOK_DELIVERER
* Use decorator for /api/user PAK auth
* Don't fire REST hook if user doesn't have "zapier" feature
* Import Optional
* Reword to "premium Zapier"
* Make mypy happy
* Fix test_delete_personal_api_key
* Fix misclick
* Fix and test /capture with personal API key
* Make mypy happy
* Remove extra_properties_json
* Resolve migrations
* Remove apt-utils
* Optimize and test PAK user.is_active filtering
* Replace DEBUG true with 1
* Remove unused instance_id
* Improve typing
* Fix deletion toast
* Refactor CopyToClipboard and use it in PAKs
* Use toast.success
* Update migrations
* Fix migration
* Fix migrations
* Complete merge
Co-authored-by: Tim Glaser <tim@glsr.nl>
